Data Vault MOdelling

Data Vault Data Modelling belongs to the family of Ensemble Data Modelling.

In discussions about modern data warehousing, terms like Data Vault (DV), Business Vault (BV), Raw Vault, and Information Vault are often used, leading to confusion. However, it’s crucial to clarify there is only one Data Vault methodology, designed to streamline and enhance data warehousing. Originating in 2000, Data Vault was developed as a comprehensive, agile warehousing method that effectively manages complex data requirements. What some refer to as “Business Vault” or “Raw Vault” are not distinct systems but extensions within the core Data Vault model, designed to address specific needs within the same foundational framework. This misconception often creates unnecessary complexity; it’s more accurate to simply refer to “Data Vault” as a single, unified approach.

To fully understand Data Vault modelling, it's helpful to first revisit foundational Data Warehouse concepts. Decades ago, Bill Inmon, known as the "Father of Data Warehousing," introduced the Data Warehouse concept, based on third normal form (3NF) data modelling, which emphasizes organizing data by business entities to reduce redundancy and maintain integrity. Ralph Kimball later popularized the dimensional approach, which uses star or snowflake schemas to simplify data for analytical purposes. This article will focus on the Data Vault model, which is a modern, agile alternative that addresses some of the limitations of traditional models.

Building a Holistic Data Architecture

Data architecture generally consists of several key zones, each serving a specific role in the data lifecycle. These zones include:

• Landing Zone: This is where raw data from various sources enters the architecture.
• Staging Zone: Here, data undergoes initial processing, including cleansing and formatting, in preparation for more complex transformations.
• Transform Zone: Often divided into two layers:
  • Warehousing Layer: Traditionally uses Inmon's 3NF approach to organize core business entities.
  • Semantic Layer: Applies Kimball's dimensional modelling to enable analytical queries, often through star or snowflake schemas.

In this setup, I see 3NF modelling as residing within the Transform Zone's Warehousing Layer, while dimensional modelling resides in the Semantic Layer. Data Vault modelling, as an alternative, replaces 3NF in the Warehousing Layer, introducing flexibility and scalability to support agile data warehousing.

The Rationale Behind Different Data Modelling Approaches

Before diving into the specifics of Data Vault, it’s useful to understand the principles behind 3NF and dimensional modelling, as each offers distinct advantages:

• 3NF Modelling: 3NF organizes data around core business entities (e.g., customers, products) and enforces strict normalization rules to minimize redundancy and maintain data integrity. Data is stored in separate tables for transactional, master, and reference data, reducing duplicate information across tables. This model provides a foundation that is reliable and reduces storage overhead, but it can require complex joins that affect query performance.
• Dimensional Modelling: Kimball’s approach emphasizes organizing data around business facts and transactions in fact tables, with context (such as customer or product details) in dimension tables. This structure optimizes data for reporting and analysis, making it more intuitive for users and improving query performance. Although dimensional modelling is efficient for analytics, it may not be as adaptable to rapid changes or complex transactional data.

Both models serve distinct purposes: 3NF prioritizes data consistency and integrity, while dimensional modelling simplifies data access for analytics. When implementing Data Vault, dimensional modelling in the Semantic Layer can still be valuable, enhancing reporting capabilities without disrupting the modular structure of Data Vault.

Data Vault: A Flexible, Scalable Approach

Data Vault offers a modern, agile approach to data warehousing, addressing some limitations of traditional 3NF models by introducing three main table types:

• Hub Tables: These tables store business keys, which are unique identifiers for entities (e.g., Customer ID or Product ID). Hubs organize data by core business concepts without storing descriptive information.
• Link Tables: These tables manage relationships between entities, capturing connections through foreign keys that link hub tables. Links allow the data model to evolve as relationships between entities change or new relationships are discovered.
• Satellite Tables: These tables contain descriptive attributes related to hub entities. By separating contextual information (such as customer names or product details) from the core keys in the hub, satellites make it easier to update or expand descriptive information without affecting relationships or business keys.
• Reference Tables (Optional but Recommended)

Reference tables store stable domain values such as loan status codes or product categories. While Satellites can store raw codes directly, Reference tables are often preferred for shared or standardized domain values, ensuring consistency and reducing redundancy across the warehouse.

Podcast

Each entity forms its own mini model within the Data Vault structure, consisting of a Hub, Link, and Satellite table(s). For instance, a "Customer" entity would have:

• A Hub Table for unique customer identifiers.
• One or more Satellite Tables for attributes like name and address.
• A Link Table to connect with its Hub table of the customer entity, along with other entities, such as "Orders" or "Accounts."

Integrating Small Models into a Larger Ecosystem

A distinctive feature of Data Vault is how it interconnects these small models to create a larger, flexible data ecosystem:

• Modular Growth: Since each entity forms a distinct mini-model, new entities and relationships can be added independently, preserving data integrity and continuity.
• Flexible Expansion: Link Tables enable relationships between entities without altering core data, supporting agile responses to changing business requirements. Hub tables from one model can connect to Link tables from another, creating a network of interconnected data that grows with the business.

This modular approach allows Data Vault models to scale in response to business needs, making it an ideal choice for complex, dynamic environments. The agility inherent in Data Vault modelling supports rapid development cycles, allowing teams to adapt data structures over time without extensive rework.

Incorporating a Semantic Layer for Analytics

Data Vault's strength lies in its agility and scalability, but complex queries can require significant processing, particularly as the data grows. For this reason, a Semantic Layer, often in the form of dimensional models or physical tables, can be incorporated to optimize analytics and improve performance. Options for adding a Semantic Layer on top of Data Vault include:

• Virtual Semantic Layer: This approach applies on-the-fly joins across hub, link, and satellite tables, preserving Data Vault's flexibility but potentially increasing query complexity.
• Physical Tables: For frequently accessed data, a physical Semantic Layer can pre-aggregate complex data queries, enhancing performance by reducing query load.
• Dimensional Modelling: A dimensional model within the Semantic Layer simplifies reporting for end-users, providing a structured way to view data while leaving the Data Vault structure intact in the Warehousing Layer.

These Semantic Layer options ensure that a Data Vault architecture remains agile without compromising on analytics, delivering a balance between flexibility and performance.

Benefits and Applications of Data Vault

Data Vault is particularly valuable for organizations that need a high degree of flexibility, scalability, and agility in their data warehousing solutions. It addresses challenges associated with traditional models by:

• Adapting to Change: Data Vault’s modular structure supports iterative development, making it easier to add or adjust data sources, entities, and relationships without extensive rework.
• Supporting Data Consistency: The separation of business keys, relationships, and descriptive information enhances data quality and consistency across the data warehouse.
• Improving Scalability: By breaking data down into Hubs, Links, and Satellites, Data Vault supports scaling at both the entity and relationship levels, allowing organizations to grow their data architecture as their business evolves.

Data Vault is therefore well-suited for complex data environments, particularly those requiring a flexible data architecture that can accommodate rapid changes and frequent updates. It enables agile development, scalable data integration, and maintains a foundation of data integrity and accuracy.

In conclusion, Data Vault offers a unique approach that combines the consistency and normalization of 3NF modelling with the flexibility and modularity needed in dynamic business environments. By implementing a Data Vault architecture with a supportive Semantic Layer for analytics, organizations can maximize the benefits of their data infrastructure, building a data warehouse that is both resilient and agile in the face of change.

“In Data Vault Modelling, every satellite table is temporal by design: ValidFrom/ValidTo capture business reality, and LoadDate captures when the system recorded it. All three fields together make Data Vault Modelling a bi-temporal framework by default, not by exception.”

Loan Approval Example (Data Vault Structures)

• Hub_Loan → {LoanID}
• Hub_Customer → {CustomerID}
• Hub_Officer → {OfficerID}
• Link_LoanCustomer → {LoanID, CustomerID}
• Link_LoanOfficer → {LoanID, OfficerID}
• Sat_LoanDetails → {LoanID, LoanAmount, InterestRate, StatusCode, ValidFrom, ValidTo}
• Ref_Status (Optional) → {StatusCode, StatusDescription}

So, if a loan’s interest rate changes from 7.5% to 8.0%, the old value stays in the Satellite with its validity range, and the new row is added, preserving full history.

How it Reads

• Hubs: define unique business keys (Loan, Customer, Officer).
• Links: connect the entities (Loan → Customer, Loan → Officer).
• Satellites: capture all changing attributes with full historization.
• Reference (Optional): Lookup tables for stable domain values (e.g., loan status codes)

So, if regulators ask, “Who was the officer, and what interest rate was valid as of Jan 20, 2024?”, you can reconstruct it directly.