Knowledge Hub

A Data Dictionary is the next step towards mitigating the Risk of a potential Data Swamp.

‘Data Dictionary comes under Metadata Management.’

In the IT World, we all know that new products and solutions keep coming year by year, so it's very important to capture what has been implemented. A Data Dictionary is one of the most important items for a system's impact analysis as it gives a clear picture of all the elements in the database.

With the introduction of Big Data where semi and unstructured datasets have come onto surfaces like video, audio, logs, emails, pdf & social media etc., so if one doesn't have a Data Dictionary in place then all future implementations will take 2x-3x more time.

Data Dictionary stores metadata information about databases like.

• List of all the objects and when each was created or updated.
• Details like field name, description, data type, length, indexes, primary & foreign keys etc.
• Relationships between tables, schema(s) and databases etc.
• Number of records in each table

We discussed the Data Dictionary and the Data or Business Glossary. What is Data Catalog?

‘Data Catalog comes under Metadata Management.’

Although all three Data Dictionaries, Data or Business Glossary and Data Catalog look similar, it's very important to understand the difference.

Data Dictionary stores metadata details about the solution and Data or Business Glossary is the collection of all business terms about the solution whereas Data Catalog jointly refers to Dictionary and Glossary like which business term is referring to which table or dashboard etc. In other words, Catalog gives a complete picture of both Dictionary and Glossary.

Let's take an example of a Bank Account Number (BAN).

• Data Glossary: BAN is the unique identifier for a customer across all banking applications whether it’s HR, Finance, Marketing etc.
• Data Dictionary: BAN is a unique identifier with field type as string, field length as 10, with two times # in it etc., metadata detail.
• Data Catalog: It tells the whole story i.e., BAN is the unique identifier for a customer across all banking applications and it is stored in tables i.e., x, y, and z, it is in which schema and database. And which department owns which databases, tables, dashboards, and reports etc.

In today’s hyper-competitive digital landscape, the imperative for a robust Enterprise Architecture (EA) framework cannot be overstated. For Top Management, synchronizing technological initiatives with overarching business imperatives is essential to optimizing value creation and operational efficacy.

A holistic EA framework initiates with the essential inquiry of "WHY?", the foundational rationale underpinning each initiative. By accurately revealing the purpose and securing explicit management endorsement, each project is thoroughly aligned with strategic imperatives. Comprehensive guidance is then provided on establishing precise Go-Live timelines, delineating resource requisites, and selecting optimal deployment paradigms, whether on-premises or within a Cloud ecosystem.

A robust EA framework encapsulates four pivotal architectural dimensions. Business Architecture is devoted to harmonizing business exigencies with strategic aspirations, facilitating the development of cogent business strategies, models, and processes that engender substantial value for stakeholders. Data Architecture is predicated on the meticulous organization and management of data, fostering the creation of sophisticated data models, definitions, and flows that yield actionable insights and inform strategic decision-making. Application Architecture addresses application requisites, ensuring that enterprise systems and databases are finely tuned for peak performance and functional integrity, thereby augmenting value for end-users. Lastly, Technical Architecture concentrates on the deployment of requisite physical infrastructure, fortified security protocols, and advanced network solutions to support the technical landscape and guarantee the seamless execution of strategic initiatives.

One must first understand the existing Enterprise Data Architecture Landscape as every new project’s data architecture must follow existing Landscape rather introducing new components with every new project. I have personally seen organizations creating Data Components/ Tools Swamp, as in current era of Big Data there are hundreds of tools offering the same functionalities so one must know whether a tool is already there in existing Landscape or not for a particular requirement. I would consider this as, one the biggest Risk in current era.

Data Architecture is the most important aspect of any data related solution or project. As mentioned in the previous chapter, there are four types of architecture i.e., 1) Business Architecture 2) Data Architecture 3) Application Architecture and 4) Technical Architecture. Most of the organizations confuse or mix-up the last three, which is again a recipe for disaster. Data Architecture should remain standalone from Application or Technology Architectures.

Medallion Architecture is a new term but again just like many other jargons used by Salespeople, this one is no more than a Buzz word. In other words, we already know this, we are already use it if we have a lakehouse implemented.

People who are working with Databricks and have implemented their Lakehouse probably know what this Medallion Architecture is. Let me try to explain in a few lines for the audience reading this book.

Medallion Architecture is sitting in Delta Lake tables, and it is about processing data in multiple layers i.e., Bronze ⇒ Silver ⇒ Gold layer tables.

• Raw Layer: This layer is normally not under Medallion Architecture as it totally Raw whereas Bronze, Silver and Gold all are in Delta Lake tables. So Raw Layer hosts source data which can’t be pushed to Bronze Layer directly. Most of the organizations prefer to land all source systems data in Raw layer (I also prefer this).
• Bronze: Those source systems which have connectors with Databricks/ Delta Lake can land data in Bronze else Bronze source is most Raw layer. Some data standardization transformation happens here like.

Framework for Big Data Processing

Lambda Architecture is an essential concept in the world of Big Data, gaining attention as technology for handling massive data volumes has advanced. It forms the backbone of many decision-support systems by providing a structured approach to data processing, integrating both batch and real-time data workflows. The Lambda Architecture model offers a comprehensive system for handling data from generation to end-user access, balancing speed and accuracy through batch and real-time layers.

The Foundations of Lambda Architecture

Lambda Architecture is grounded in lambda calculus, a formal system in mathematical logic that supports computation using function abstraction, application, and variable substitution. Lambda calculus is a universal model that can theoretically simulate any Turing machine, making it foundational to computer science and software engineering. In data processing, Lambda Architecture uses this mathematical basis to create a dual-layered system that can handle data asynchronously and in real time.

Kappa Architecture is mostly confused with Lambda Architecture which is fair as both processes Streaming as well as Batch data. Then how both are different?

Due to high demand of Real-Time data, Kappa Architecture has emerged but not for all kinds of data sizes just like Lambda Architecture is.

We have gone through in detail (in separate topic) like what is Lambda Architecture so won’t go in details like how Streaming and Batch works. Let’s understand how Kappa is different than Lambda Architecture.

Kappa Architecture offers ONLY stream processing using Queuing tools like KAFKA for both Streaming and Batch datasets. It doesn’t offer separate tools or technologies to process batches. It says, process both Real-Time Streaming and Batch data using same tool as KAFKA but the biggest differentiator here is, it stores data in a separate data store which might not be suitable to store TBs or PBs of dataset. It offers batch related downstream applications to connect directly to its storage.

Unified Data Architecture is about covering all aspects of data with respect to organizational architectures. There is no pre-defined Unified Data Architecture for everyone.

Let’s see how many Architectures we can have, in an organization with respect to Data.

• Architecture for Source Applications i.e., OLTP
• Architecture for Data Warehouse i.e., OLAP and Cubes
• Architecture for Data Lake, Delta Lake, Data Mesh, Data Vault, Business Vault, Data Hub, Data Fabric
• Architecture for Advance Analytics i.e., for Machine Learning and Deep Learning etc.
• Data Virtualization Architecture

As highlighted above, if not all but there can be many Data Architectures in-place in every organization. In the current era, where data has become the central focus element for everyone, one must design Unified Data Architecture for 30,000 feet view before designing Data Architectures for each component mentioned above.

Just like Enterprise Architecture is about Business Architecture, Application Architecture, Technical Architecture and Data Architecture, all these must follow Enterprise Architecture standards. Same way, all Data Architectures in an organization must follow Unified Data Architecture standards.

In today’s rapidly evolving digital landscape, security threats are more sophisticated and pervasive than ever. Traditional perimeter-based security is no longer sufficient to protect your organization’s most valuable asset – its data. This is where Zero Trust Architecture (ZTA) comes into play.

Data breaches can have catastrophic consequences, both financially and reputationally. Zero Trust Architecture ensures that no entity – human or machine – is trusted by default, minimizing the risk of unauthorized access. With millions of IoT devices and diverse infrastructures spanning on-premises and cloud environments, ZTA offers a robust security framework that addresses these complexities through stringent authentication, authorization, and entitlement mechanisms.

Starting with a zero-trust baseline where no one has access, the architecture gradually opens doors through verified credentials and continuous monitoring. This adaptive approach, combined with data encryption, masking, and anonymization, ensures that your defenses are resilient against emerging threats. Implementing Zero Trust Architecture significantly reduces the risk of data breaches, ensures regulatory compliance, streamlines security operations, and future-proofs your security strategy.

NEON-Inspired Architecture refers to a modern architectural approach that separates storage from compute, allowing systems to scale more efficiently and flexibly. The idea is inspired by cloud-native database designs where data storage and processing power are decoupled instead of tightly integrated.

In traditional database systems, storage and compute are bundled together. If you need more processing power, you often have to scale the entire system, including storage, even if storage capacity is sufficient. This creates inefficiencies in cost and scalability.

In a NEON-inspired approach, storage and compute operate independently. Storage is centralized and durable, while compute instances can scale up or down based on workload demand. This architecture supports elasticity, better resource utilization, and faster provisioning of environments.

One of the well-known implementations of this concept is Neon, which applies serverless principles to PostgreSQL by separating compute nodes from storage layers. While the term “NEON-inspired” may not refer exclusively to that company, it reflects the broader architectural pattern of disaggregation and cloud-native database design.