My data engineering experience dates back 4 years ago, and it is closely related to Apache Spark, Hadoop, and Hive. Within those 4 years, I am amazed at how quickly the ecosystem has evolved.

What Captured My Attention…

Increases in Interoperability Between Data Formats

• Databricks Agrees to Acquire Tabular: Tabular is a data management platform based on Apache Iceberg, founded by the creator of the Apache Iceberg Data format.

• General Availability of the Delta Lake Universal Format: Databricks also emphasized their desire to bridge the gap between Apache Iceberg and the Delta Lake storage layer. UniForm is a push in this direction.

• Databrick's push for Delta Kernel: Delta Kernel is an initiative to simplify the creation and use of Delta connectors. The user can then leverage the Table API and an Engine API to access its data.

Various Ways to Integrate Between Organizations and Clouds

• Unveiling of LakeFlow: A solution for data engineering with capabilities such as unified integrations of data sources and batch/streaming transformation of pipelines. Currently on private preview, and only supports datasource integrations.

• Clean Rooms: A solution that allows secure data sharing and collaborative experiments between organizations.

• Delta Sharing Protocol: Used to share data between organizations and cloud platforms.

Other Interesting Discoveries…

• Shutterstock offers its own AI image generator. This way, users avoid intellectual property concerns since Shutterstock owns all the images.

• Stored Procedures are seen as a major technical debt when migrating warehouses to Databricks: https://medium.com/dbsql-sme-engineering/top-10-reasons-why-your-cloud-data-platform-migration-should-leave-stored-procedures-in-the-past-ceb0a037bd55

• Application of the Data Vault Pattern inside the Medallion architecture for PII protection and GDPR compliance: https://www.databricks.com/dataaisummit/session/ensuring-gdpr-compliance-real-time-data-pipelines-conde-nast

• Select Star

• Glean

Images…

Delta Universal Format (UniForm):

Delta Kernel:

Delta Sharing:

LakeFlow - LakeFlow Connect