Knowledge Hub

NoSQL Data Modelling refers to the practice of designing data structures for non-relational databases such as document stores, key-value stores, wide-column databases, and graph databases. Unlike relational modelling, which is driven by normalization theory and functional dependencies, NoSQL modelling is driven primarily by application access patterns and workload requirements. The goal is not theoretical purity but scalable, high-performance data access.

The primary objective of NoSQL data modelling is to support flexible or schema-less storage, distributed scale-out architectures, and fast, query-driven design. Instead of decomposing data into highly normalized tables, NoSQL models are often intentionally denormalized or embedded to minimize joins and reduce cross-node communication in distributed environments. Data is structured according to how it will be read and written, not according to relational dependency rules.

Document databases store data as JSON-like documents that can contain nested fields, arrays, and variable structures. Instead of organizing information into rows and columns, they treat each document as a self-contained representation of a business object. The goal is to provide flexible schema design and rapid development by allowing complete entities to be stored and retrieved in a single record.

In a document database, each document typically represents one business object. In a loan approval system, for example, a single document may represent an entire loan application. Customer details, loan officer information, approval status, and repayment history can all be embedded within that same structure. Because the schema is flexible, new fields can be added without requiring costly table migrations or rigid schema alterations. This makes document databases particularly attractive in fast-moving development environments.

Key–Value databases represent the simplest form of non-relational data storage. They store data as a pair consisting of a unique key and its associated value. The key serves as an identifier, while the value contains the data itself. This model eliminates structural complexity and focuses entirely on fast retrieval by key.

The primary goal of key–value databases is to deliver ultra-fast lookups with minimal overhead. They are designed for workloads where the access pattern is simple and predictable, such as caching, session management, token storage, or high-volume read/write operations. Instead of supporting complex queries, these systems optimize for speed and scalability.

In a key–value database, each record is accessed directly through its key. The value associated with that key can be a simple string, a JSON object, or even a binary blob. There is no concept of joins, relational constraints, or query-based filtering beyond key lookup. This makes the model extremely efficient for operations such as “fetch loan details by LoanID.”

Wide-Column databases, often referred to as column-family databases, organize data into rows and dynamic columns grouped into column families. Unlike traditional relational tables where every row follows a fixed schema, wide-column systems allow each row to contain a flexible and potentially sparse set of columns. They are optimized for distributed storage and partitioned queries across very large datasets.

The primary goal of wide-column databases is to enable high write throughput and scalable query performance over massive volumes of data. They are particularly well suited for time-series, event-driven, and high-velocity workloads where large amounts of data must be written and retrieved efficiently across distributed clusters.

In this model, data is partitioned by a primary key, for example, CustomerID or LoanID. All related data for that key is stored together within a partition. Within each partition, rows may contain flexible columns that vary between records. The schema is shaped primarily by access patterns. Instead of normalizing entities into separate tables, designers structure data to answer specific queries efficiently.

Graph databases store data as nodes and edges, optimized specifically for representing and querying connected information. Nodes represent entities, while edges represent relationships between those entities. Unlike relational systems that organize data into tables and connect them through joins, graph databases treat relationships as first-class citizens, enabling direct traversal across connections.

The primary goal of graph databases is to uncover insights from relationships and networks. They are particularly effective in scenarios such as fraud detection, influence analysis, recommendation systems, and complex workflow analysis, areas where relational joins become expensive and difficult to manage at scale. When the question being asked is fundamentally about how things are connected, graph databases often provide a more natural solution.

In a loan approval context, nodes might represent entities such as Customer, Loan, LoanOfficer, and Branch. Edges define how these entities are connected. For example, a Customer applies for a Loan, an Officer approves a Loan, and a Loan is issued at a Branch. Because relationships are stored explicitly rather than inferred through joins, queries can traverse the network efficiently.

NEON-Inspired Architecture refers to a modern architectural approach that separates storage from compute, allowing systems to scale more efficiently and flexibly. The idea is inspired by cloud-native database designs where data storage and processing power are decoupled instead of tightly integrated.

In traditional database systems, storage and compute are bundled together. If you need more processing power, you often have to scale the entire system, including storage, even if storage capacity is sufficient. This creates inefficiencies in cost and scalability.

In a NEON-inspired approach, storage and compute operate independently. Storage is centralized and durable, while compute instances can scale up or down based on workload demand. This architecture supports elasticity, better resource utilization, and faster provisioning of environments.

One of the well-known implementations of this concept is Neon, which applies serverless principles to PostgreSQL by separating compute nodes from storage layers. While the term “NEON-inspired” may not refer exclusively to that company, it reflects the broader architectural pattern of disaggregation and cloud-native database design.

Data Observability is one of the most underrated terms, as its name says, it's about observing and providing analysis and analytics for decision making.

‘Data Observability comes under Metadata Management.’

Yes, Data Observability aims to focus on observing every data pipeline, data transformation, DDL, DML, data governance standards, security & privacy policies etc. across the organization. It creates a holistic view of everything, it assists to find out where the problem is and debugging steps assist to make future decisions, so the same problem don't occur in future.

Pillars of Data Observability

1. Freshness
2. Lineage
3. Schema
4. Volume
5. Distribution

Now the question is, observe but how? Do we or can we give access to one framework on all the frameworks in Enterprise? Of course, the Security, Risk, Compliance and Audit departments won't let it happen. But NOT-IF the access is requested only on tools for Metadata Management. Yes, Data Observability mainly access Enterprise Metadata Data.

Data Preservation, Data Security, Data Privacy, Data Integrity, Data Compliance all refer to the protection of the data.

Data Preservation is about securing and maintaining the safety and integrity of the data after it’s been processed, for future reference.

‘Data which is preserved cannot be changed and hardly purged or deleted.’

Things to consider for Data Preservation

• What data to be preserved? Not all data is worth preserving.
• What is the retention period to preserve data? Like for how long to keep data.
• Who will be responsible for preserving data? A custodian is required for data repository.

In the current era with big data having huge semi and unstructured datasets, it’s not worth rather not possible to preserve everything. One must decide what to preserve and for how long.

For example, in datastore like Data Warehouse or a Data Mart or in Data Lake etc., there are certain datasets to preserve. One can start from the list of compliance requirements e.g., traditionally in banking sector, the customers who has left the bank or in other words closed their accounts, their history is preserved or saved for 7 years from the day they closed their accounts.

Data Subsetting is as simple as taking a small piece of data, with referential integrity in place, for some activity. For example, there is a requirement to build a data science model and it requires data for users aged between 25-35 from X Location having certain buying/ spending patterns. For this requirement, a particular dataset will be extracted for the data science development team to build and train their data models.

Don't confuse Data Subsetting with Data Masking. Data Masking is still required on this smaller piece of the dataset from a security point of view. Having said that, conducting a Data Subsetting exercise is not as simple. One first must understand the functional requirements of the use case, then one must also understand what data is there in the database as whole. Wrongs or partial Data Subsetting will end up producing wrong data model results and will damage trust in data by decision-makers.

One of the biggest criticisms of early Data Lakes was their lack of reliability. While they excelled at storing vast volumes of structured, semi-structured, and unstructured data, they struggled to provide the transactional integrity and data consistency that relational databases and warehouses delivered as a matter of course. Operations such as inserts, updates, and deletes, known collectively as DML (Data Manipulation Language), were difficult to perform reliably on file-based storage systems like Hadoop Distributed File System (HDFS) or cloud object stores such as Amazon S3.

To achieve performance and flexibility, many organizations turned to Apache Spark, an in-memory processing engine capable of handling both batch and real-time streaming workloads. However, Spark introduced a different problem: while it enabled fast queries and analytics, it was not ACID-compliant. Without atomicity, consistency, isolation, and durability, organizations faced the risk of incomplete writes, corrupted data, or inconsistencies when jobs failed midway. Workarounds existed, but they were complex and fragile.