Knowledge Hub

Visionaries and business users thought that because they understand their business, know their problems, understand their use cases, and now have Vibe Coding and AI-assisted development at their fingertips, they could build things themselves.

So they began asking:

"Why do we need Data people? We understand the business. We know what we want to build. And AI can help us build it."

And to be fair, the early results look promising.

POCs went well.
Pilots went well.
Applications were built. Dashboards were created. Workflows were automated.

Everyone is excited.

But now comes the interesting part.

• Now it's time to bring in the actual enterprise data.

• Now it's time for proper user acceptance testing.

• Now it's time to fix data quality issues that nobody noticed during the POC.

• Now it's time to optimise the code because compute consumption is increasing, cloud costs are approaching budget limits, and the expected ROI is taking a hit.

• Now it's time to optimise storage, remove unnecessary data duplication, and manage growing volumes.

• Now it's time to integrate with existing enterprise systems, deal with incompatible data structures, and resolve dependencies.

• Now it's time to migrate from legacy technologies to newer platforms without disrupting business operations.

• Now regulatory bodies are asking questions. We need to produce compliance reports, demonstrate data lineage, maintain audit trails, and prove where the data came from and how it was transformed.

• Now business leaders want transparent and trusted KPIs. We need to reconcile different definitions, validate calculations, and ensure everyone is looking at the same numbers.

I have seen organisations invest years building data platforms, introducing new technologies, and expanding their enterprise architecture. Over time, these investments become part of the technology landscape.

Then something interesting happens.

A new platform is introduced, but the old one stays. A new data warehouse is built, but the previous warehouse continues running. A new reporting solution is launched, but business users still depend on the old reports. Another integration layer is added because replacing the existing one is considered too difficult.

Eventually, the organisation is operating multiple generations of technology, often solving similar problems in slightly different ways.

And nobody really wants to switch anything off.

There are understandable reasons. Dependencies are complicated, business users are comfortable with existing systems, migration carries risk, and nobody wants to be responsible for an outage.

Enterprise data architecture is failing. Not because organizations lack investment, or because the problem is inherently too complex, but because the structure required to hold these ecosystems together is either weak or absent. As data landscapes stretch across domains, platforms, and workloads, coherence simply does not sustain itself. Below diagram show the potentiate challenges faced by different persona(s) in an organization.

Over the past decade, there has been no shortage of progress. Cloud-native platforms, lakehouse patterns, distributed processing, governance tooling - none of this is immature anymore. And yet the outcomes tell a different story. Environments expand, but alignment erodes. Governance becomes more visible, more documented, more discussed - but less effective in practice. Analytical capability grows, but trust lags behind, sometimes significantly. These are not isolated execution failures. They point to something more fundamental.

Data has become one of the most powerful words in our modern vocabulary. It drives our world, impacting decisions in industries, governments, and our daily lives. But what exactly is data? How do we define it in a way that captures its full impact and utility?

Mustafa Qizilbash
“Features and Activities of an Object is Data.”

This phrase encapsulates data’s essence. But let’s break down what it really means.

Origins and Traditional Definitions

The word "data" originates from the Latin word datum, which means "something given". This concept persists in how we perceive data today: it’s a foundational input, a starting point. In French, for example, data is referred to as données, highlighting its roots as something "given" or granted for analysis.

If you look up data in Wikipedia, you might find it defined as "individual facts, statistics, or items of information, often numeric." More formally, data is a set of values related to qualitative or quantitative variables about one or more entities or objects. In this sense, each "datum" represents one specific value or measurement.

By W H Inmon

Everywhere you read, every news show you watch, every ommercial you see there is this overwhelming fascination and fixation with AI. The AI vendors have sold the world on the idea that AI is the future of mankind.

In a word, people and the world are stoked.

NEW POSSIBILITIES

And what is it about AI that gets people so excited? There are many facets of AI that appeal to people. AI can talk with you. AI can create presentations. AI can answer questions for you. AI can look things up and find a multitude of facts – both obscure and contemporary.

The interface, the capabilities, the range of possibilities of AI are just staggering. And people have never seen this before.

AI has opened up a whole new world.

Now, people that have never touched a computer before can do sophisticated analysis on the computer.

SELLING AI

And who has AI been sold to? AI has been sold at the highest levels of society – to the board of directors, to politicians, to decision makers everywhere. Unlike previous technologies that have been sold to technicians, AI has been sold at an entirely different level.

In the world of data and databases, the term entity is foundational yet often misunderstood. To build reliable data systems, it is essential to understand what an entity is and how it is structured. This chapter explores the concept of an entity and breaks it down into its three core components: Entity Type, Entity Attributed and Entity Instance.

Let’s decode it.

What Is an Entity?

An entity is anything that can be uniquely identified and described. In data modelling, an entity typically refers to a real-world object, person, concept, or event that holds relevance within a system, such as a student, a book, or a transaction.

Entities are the fundamental building blocks of structured data models. To model and manage them effectively, we categorize entities into two levels:

• Entity Type: the blueprint or template
• Entity Attribute: the properties
• Entity Instance: the real-world, data-filled object

1. Entity Type – The Blueprint

Entity Recognition, commonly known as Named Entity Recognition (NER), is a technique used to identify and extract meaningful entities from unstructured text. It is also referred to as entity identification or entity extraction. While entity recognition has long been appreciated in data modeling, particularly during conceptual modeling when identifying core business entities, it plays an equally critical role in the data science domain.

In traditional structured systems, entities such as Customer, Product, Order, or Invoice are clearly defined within database schemas. However, modern organizations generate enormous volumes of unstructured data from social media posts, emails, PDF files, Word documents, comment columns, chat transcripts, and even video or audio content converted into JSON or text formats. For data modelers, this unstructured layer represents one of the greatest challenges when designing robust data models.

Primary Key is also known as Pkey, and Foreign Key is also known as Fkey. These are one of the most important terms without which a proper Data Model cannot be created.

We have understood what a table is. Now, the next step is to understand, if there is more than one table then how to join both. In the topic ‘Tables, Columns and Rows’ we used an example of table ‘Family’. Let’s use the same table here with an additional column i.e., Birth Location.

Table Name: Family

Now let’s say, to save space in Family table as we can see ‘United State of America’ has been used many times so how about creating another table where let’s store lengthy country names and in Family tables, let use country code instead with there is another table called as Birth Location.

Table Name: Birth Location

Now, let’s update the Family table to reflect country code instead of country name.

Table Name: Family

Now, the problem is column ‘Birth Location’ have country code so how to find out which country name refer to ‘001’. Let’s declare ID in Birth_Location table as Pkey and Location_ID in Family table as Fkey.

In data management, there are three fundamental data types: structured, semi-structured, and unstructured. These data types can originate from numerous sources, including databases, mobile applications, social media platforms, sensors, logs, CCTV footage, radio signals, and more. While structured and unstructured data are widely recognized, the classification of semi-structured data often leads to debate.

Structured Data

Structured data refers to information organized in a predefined format, typically consisting of rows and columns. For instance, imagine an “employee-contact-details” table containing fields such as employee code, email, mobile, LinkedIn URL, Instagram, and Twitter. Although not every employee will have complete details, the table structure remains intact when exported, with each row containing a set number of columns, even if some fields are empty. For example, a dataset with 6 columns and 10 rows will consistently display this format, regardless of missing data entries.

Good data in the data world possesses several essential qualities that contribute to its reliability and effectiveness. Firstly, it is accurate, ensuring that the information it represents aligns with the real-world context it intends to portray, free from errors or inconsistencies. Completeness is another hallmark of good data, implying that it includes all necessary details without any missing or incomplete records. Consistency is maintained throughout, following standardized formats and structures, facilitating uniformity and coherence. Relevance ensures that the data is directly applicable to its intended purpose, meeting the specific goals of data analysis or application. Timeliness is a key aspect, as good data is up to date, reflecting the most recent and relevant information. Reliability is associated with trustworthiness, ensuring the data is unbiased and collected through reliable methods. Validity is maintained by adhering to predefined rules and standards, and accessibility ensures that authorized users can easily retrieve and utilize the data.