Medallion Architecture

Medallion Architecture is a new term but again just like many other jargons used by Salespeople, this one is no more than a Buzz word. In other words, we already know this, we are already use it if we have a lakehouse implemented.

People who are working with Databricks and have implemented their Lakehouse probably know what this Medallion Architecture is. Let me try to explain in a few lines for the audience reading this book.

Medallion Architecture is sitting in Delta Lake tables, and it is about processing data in multiple layers i.e., Bronze ⇒ Silver ⇒ Gold layer tables.

• Raw Layer: This layer is normally not under Medallion Architecture as it totally Raw whereas Bronze, Silver and Gold all are in Delta Lake tables. So Raw Layer hosts source data which can’t be pushed to Bronze Layer directly. Most of the organizations prefer to land all source systems data in Raw layer (I also prefer this).
• Bronze: Those source systems which have connectors with Databricks/ Delta Lake can land data in Bronze else Bronze source is most Raw layer. Some data standardization transformation happens here like.

1. If date format is coming in different formats from different source systems, it can be standardized in this layer.
2. NULL values can be given some values.
3. Data types and columns sizes can be fixed.
4. CDC can be performed at this layer to extract delta data/ incremental data.

Please note, at this layer, data models remain the same vs source systems.

• Silver: It is where data is transformed in Enterprise Data Warehouse, it’ not mandatory but in Bill Inmon 3-NF data warehouse model.
• Gold: It is where Data Marts or Semantic Layers comes into picture focusing business KPI(s) like Customer Analytics, Product Quality Analytics, Inventory Analytics, Customer Segmentation, Product Recommendations, Marking/Sales Analytics etc.

Please note, we all have been working with above methodologies in past like we used or still have layers i.e., Landing 🡪 Staging 🡪 Data Warehouse/ Data Mart 🡪 Semantic/ Aggregation Layers 🡪 Cubes. The only difference is Databricks has brought everything in Data Lake eliminating RDBMS(s).