Knowledge Hub

Yes, you are right, you might have not heard, or implemented HOOKS before. This is a new term introduced by Andrew Foad.

HOOK is a technique to Data Warehousing especially proper to Data Lakehouse and virtualized architectures, but having said that it can be applied on any SQL based database. It is a hybrid methodology that borrows features from other methodologies such as Data Vault and Kimball dimensional modelling.

Whereas Data Vault and Kimball methodologies have multiple table types (Hubs, Links, Satellites, Dimension, Facts, etc.), HOOKS has a single structure called a Bag, which is used to wrap existing data warehouse table and exposes Business Keys which align to Business Concepts define in the Enterprise Data Model (EDM).

Bags align with data ingested into the data warehouse from the operational system (source-aligned) and Business Concepts defined in the Enterprise Data Model (business-aligned).

Medallion Architecture is a new term but again just like many other jargons used by Salespeople, this one is no more than a Buzz word. In other words, we already know this, we are already use it if we have a lakehouse implemented.

People who are working with Databricks and have implemented their Lakehouse probably know what this Medallion Architecture is. Let me try to explain in a few lines for the audience reading this book.

Medallion Architecture is sitting in Delta Lake tables, and it is about processing data in multiple layers i.e., Bronze ⇒ Silver ⇒ Gold layer tables.

• Raw Layer: This layer is normally not under Medallion Architecture as it totally Raw whereas Bronze, Silver and Gold all are in Delta Lake tables. So Raw Layer hosts source data which can’t be pushed to Bronze Layer directly. Most of the organizations prefer to land all source systems data in Raw layer (I also prefer this).
• Bronze: Those source systems which have connectors with Databricks/ Delta Lake can land data in Bronze else Bronze source is most Raw layer. Some data standardization transformation happens here like.

I am sure most of us know this term but not sure how accurate you are in terms of its importance.

‘The granularity of data means, the levels of data breakdown.’

Generally, people think having all relevant details about an entity is Granularity, but the question is what level of details?

For example, details can be name, email, and contact number, date of birth, color, blood group, home address, company, role, salary, father name, mother name and so. Is this the lower level of granularity? NO, when we say the lowest level of granularity, we can break down.

• Name into First Name, Middle Name and Last Name
• Address into Apartment Number, Building Name, Street, Area, City, District, Region, Country, Postal Code etc.

In OLTP systems, granularity must go as low as possible and in OLAP systems, it can depend on the business use case.

Now, when we have established what the Granularity is, let's discuss when it's needed. Data Wrangling, Data Mining, Data Science etc., all use raw data for analysis and analytics. And when we say RAW, the lower the level of granularity can be, the better the model result will be for DSS.

As the name says itself, these are traps. If we have Chasm or Fan Traps, then we have a serious problem as these can impact the whole data model.

‘Chasm and Fan Trap lead to Cartesian Data.’

Fan Trap: In Fan Trap, the model ends up having ‘One to Many’ relationship in more than two tables e.g., One Customer having Many Orders and One Order having many Order Products which is logically fine but if physical data model is created like 1) Customer with Order ‘One to Many’ 2) Order with Order Product also ‘One to Many’ then it’s not fine. It’s a Fan Trap.

• Issue: In this scenario, Order Id (Pkey) has been added in Order Products tables as Fkey which means there will be Low Cardinality in Order products tables means there will be duplication of Product in Product’s table which is a wrong approach. In the Products table, Product ID should be unique.
• Solution: One Product can have Many Orders but not the other way. The correct approach is to have Product ID (Pkey) from Order Products table as Fkey in Orders table.

Another term, if data modelers are not aware and if data model doesn’t take care via proper Data Cardinality check and balance, data analysts are hit by Chasm and Fan Traps.

We know what Data Modelling is. We know what Data Cardinality is. Now, it’s time to understand Cartesian data which give nightmares to Finance department when they manually know the total is USD100k but when they run query, they see USD200 k 😊.

‘Cartesian Data is about writing a query on tables with ‘Many to Many (M:M)’ relationship.’

If there are two tables with ‘Many to Many’ relationship, it’s time for data modeler to introduce an intermediary table which can have unique records and then can have ‘One to Many’ relationship with previously used two tables.

If the Sales table is joined with Target tables directly based on Employee ID then surely there will Cartesian data as both Sales and Target tables can have more than 1 employee ID. But hang-on, even having intermediary table, if query is written wrongly then still data analyst can hit with Cartesian data know as Chasm Trap.

‘Data Segmentation is about dividing dataset into multiple sections based on some requirements.’

Segments are defined based on many parameters e.g., in a business, segments are defined by accessibility, actionable, profitable etc.

For example, when we go into a mall, there is a segment/section for Food Court where multiple categories of food are available with different classification like expensive or cheap food.

Steps to improve Data Segmentation

• Enrich your data to increase data quality.
• Plan and replan your data segmentation criteria(s)
• Improve real time segmentation.
• Design segment based on customer prioritization and feedback.
• Integrate with as many data elements.

In business, what you can do by having Data Segmentation

• You can create deals based on customer buying behavior.
• You can improve your products arrangement.
• You can plan your marketing strategy.
• You can analyze customer feedback.

Business Analytics has been the most demanding cycle in the current era. Business Analytics supports DSS (Decision Support Systems) and the technologies to support started from OLTP systems to OLAP systems i.e., source systems to Data Warehouses or Data Marts or Data Lakes or Data Lake Houses or Data Vault or Business Vault or Enterprise Data Hubs or Enterprise Data Fabric etc., but that’s not the complete cycle. The cycle completes once Reverse ETL is performed.

Let’s understand with an example. For example, the product sales data is loaded in a system (OLTP) then it is pushed or pulled in a Data Lake or a Data/ Business Vault or in a Data Warehouse or Data Mart or Data Lake House etc. Once data has move in these datastores then analytics are performed using Data Science practice like by running Machine Learning, Deep Learning, and many other means to analyze data for prediction or forecasting and last but not the least the data is present to Top Management for decision marking. Management decides to initiate campaigns based on the data element extracted from the whole cycle from OLTP to OLAP etc.

The idea of a Data Warehouse emerged when organizations realized that transactional systems (OLTP), while excellent for recording day-to-day operations, were never designed for heavy reporting or analytical workloads. As business leaders demanded consolidated views of performance, finance, sales, and customer behaviour, data modelers introduced the concept of a centralized analytical datastore, the Data Warehouse, to relieve OLTP systems from the burden of reporting and to provide trusted, historical insight.

It is important to clarify a common source of confusion: a Data Warehouse is not the same as Dimensional Modelling. The warehouse refers to the central repository that consolidates data from multiple source systems, while dimensional modelling is only one possible technique for structuring that repository. A Data Warehouse can be designed in different ways, depending on organizational priorities such as performance, scalability, or agility.

This topic is very close to many experienced data modelers because it is surprisingly misunderstood. The confusion usually begins with one assumption: if both systems use Third Normal Form (3NF), then they must be conceptually the same. They are not.

In data architecture, the Data Warehouse concept popularized by Bill Inmon embraces the use of 3NF. Since 3NF is also widely used in OLTP systems, many architects assume the modeling philosophy and operational behavior are identical. The structure may look similar, but the intent, behavior, and lifecycle of data are fundamentally different.

OLTP 3NF: Designed for Transactional Efficiency

In an OLTP (Online Transaction Processing) system, the objective is operational efficiency. The system must support high-volume, real-time transactions such as order placement, payments, inventory updates, and customer modifications.

The primary goals are:

• Fast inserts, updates, and deletes
• Concurrency control
• Storage optimization
• Data consistency for current state

Immutable Data Warehouse has come from immutable data. Data folks are aware of it but don’t use it more often.

‘Immutable means not changeable.’

Now, the question is how it’s relevant to us in Data World. We know when we are inserting new data in a database, at the same time we also have the option to change it by using update or delete commands which means data is mutable.

Mutable feature is a MUST TO HAVE in any database at a technology level, means, any databases or data store MUST allow to update or delete. One should or should not update or delete, can be a business requirement but cannot be technology limitation.

‘By default, Data Warehouse is an Immutable datastore.’

Immutable feature is not only about, not able to update or delete data, it’s about which data should be immutable, and which should be mutable. For this, one must understand the data.

For example, CUSTOMER dataset contains many types of data e.g., name, date of birth, parents, children, address, contact details, jobs, cars etc. In this dataset, you must have judged which information is immutable and which mutable.