Knowledge Hub

Business Modelling is the practice of understanding how an organization works as a whole. It explains how value is created, how work flows through the organization, who is involved at each step, and how decisions are made. At this stage, the focus is not on data, databases, or systems, but on the business itself. The aim is to clearly describe what the organization does and how it achieves its objectives.

The main goal of Business Modelling is to provide a complete and holistic view of the business. When this view is clear, it becomes much easier to align daily operations, management decisions, and supporting systems with the organization’s strategic goals. Without this shared understanding, systems and data solutions are often built in isolation and fail to support real business needs.

Business Modelling concentrates on the “what” and the “how” of the business. It looks at business processes, roles, responsibilities, rules, and policies. It helps answer questions such as: What activities are performed? Who performs them? What decisions are taken, and based on what rules? Because of this, Business Modelling is usually the first step before moving into information modelling and data modelling. It establishes where data is created, how it flows, and how it is ultimately used.

We discussed the Data Dictionary and the Data or Business Glossary. What is Data Catalog?

‘Data Catalog comes under Metadata Management.’

Although all three Data Dictionaries, Data or Business Glossary and Data Catalog look similar, it's very important to understand the difference.

Data Dictionary stores metadata details about the solution and Data or Business Glossary is the collection of all business terms about the solution whereas Data Catalog jointly refers to Dictionary and Glossary like which business term is referring to which table or dashboard etc. In other words, Catalog gives a complete picture of both Dictionary and Glossary.

Let's take an example of a Bank Account Number (BAN).

• Data Glossary: BAN is the unique identifier for a customer across all banking applications whether it’s HR, Finance, Marketing etc.
• Data Dictionary: BAN is a unique identifier with field type as string, field length as 10, with two times # in it etc., metadata detail.
• Data Catalog: It tells the whole story i.e., BAN is the unique identifier for a customer across all banking applications and it is stored in tables i.e., x, y, and z, it is in which schema and database. And which department owns which databases, tables, dashboards, and reports etc.

We have been referring to three terms i.e., Master Data Management, Metadata Management and Data Lineage, in previous topics. Master Data and Metadata Management have already been covered. Now it’s time to decode Data Lineage which is THE most important FACTOR to save Data Lake from becoming a Data Swamp. Efficient Data Lineage can’t be built without Metadata in place.

I won’t go into details for Data Swamp, there is a separate topic on it. But any data showing in your BI Solution, and you are unable to track it back to its source means you don’t have Data Lineage and you are heading towards disaster as if not now but in future you will have no idea where the data is coming from and where it’s been used.

Data Lineage is the process of tagging all the touchpoints i.e., databases, tables, columns, business roles etc., from where data has been traveled from to BI solution or downstream systems. It also shows all the ETL(s) it went through, all the data enrichments, transformations, conversions, aggregations, and all the new measures created along the way. This helps businesses to trust the completeness and accuracy of the data for their decision-making. Data Lineage provides an audit trail of data lifecycle throughout the enterprise and assists users to understand UAM (User Access Metric).

Data Provenance and Data Lineage. Logically both do the same thing but at different levels of granularity.

Data Lineage is about knowing the Data Lifecycle but with respect to databases, tables, columns, business rule/ ETL rule etc. For example, data comes from which database, table, column and went into which database, table, column with what business rules incorporated during transformation process.

Data Provenance is at a higher level, it’s about Data Lifecycle with respect to inputs, entities, systems, and processes. For example, data comes in which format, from which source system, its related to which entity, what is the process used like real-time processing, batch processing or event based etc., any analytical modelling done, any data enrichment took place, any data cleaning, data quality rules, metadata management, master & reference data management etc., took place during the data lifecycle.

These two terms are used in Machine Learning for Data Mining and Data Science models. As we all know, within Machine Learning, there are Supervised and Unsupervised Learning methods.

Data Classification/Categorization=Supervised Learning

Data Clustering=Unsupervised Learnings

Both these terms are used for patterns identification before complex algorithms come into action.

Classification is done based on the uniqueness of the data where e.g., all animals are tagged separately whereas Clustering is done based on the characteristics e.g., 1) Animals with 2 legs 2) Animals with 4 legs.

• Data Classification uses labeled data whereas Data Clustering uses unlabeled data.
• The most popular Data Classification algorithms in data mining are the K-Nearest Neighbor and decision tree algorithms.
• The two common Data Clustering algorithms in data mining are K-means clustering and hierarchical clustering.
• Data Classification output is known.
• Data Clustering out is unknown.
• Training data is provided for Data Classification.
• No training data is provided for Data Clustering.

Data Anonymization, Data Pseudonymization or Data De-identification all worked the same way and related to Data Privacy.

Data Anonymization is about removing or modifying personal information before sharing with others so those people whom the data about remains anonymized.

‘Data Anonymization is about hiding the identity of a person to be undetectable.’

For example, while implementing a data solution it is required to share data with development team, so before sharing one must make sure data is anonymized, persons details are not exposed to development team members e.g., name, contact number, address, credit card number etc.

Data Anonymization is normally enforced by Regulatory Bodies and not following can result in heavy penalties and a bad reputation for any organization.

Techniques [All these terms are covered separately]

• Data Masking
• Data Pseudonymization
• Data Generalization
• Data Swapping
• Data Perturbation
• Data Synthetic
• Data Shuffling
• Data Scrambling
• Data Encoding
• Data De-identification

Data Compliance is mostly confused with Data Quality and Data Integrity, which is somehow true, as all these terms do share some common features for ONE GOAL i.e., Data Protection.

Data Compliance is to follow certain regulations and standards of Data Security, Data Privacy and Data Integrity, set by different Regulatory Bodies around the World to protect sensitive data and to make data available in certain ways.

Regulatory Bodies set, if not all but, CERTAIN Data Security, Data Privacy and Data Integrity standards. So, having only Data Compliance measures will not protect data 100%, there are many other standards covered under Data Security, Data Privacy and Data Integrity which are required to be in-place for any organization.

Common Regulatory Bodies Frameworks

We won’t be going into details of regulatory bodies like what is required by each but for the audience, let me mention the most famous ones here.

A term very near to Data Security and Data Privacy. Data Sovereignty is about keeping the data secure within the defined boundaries, especially e.g., certain sets of data should not go cross border for certain sectors.

In Data Security and Data Privacy, we make sure data is only accessed by those who need it or who own it. Within the organization, we maintain an ACL (Access Control List) to record the list of all the users with their level of access they can have on certain data sets. For example, which level of access developers can have, which level of access database administrators can have, which level of access department’s manager can have, which level of access a Business User can have, which of access Cost Centre person can have etc. So, in a way what we are trying to elaborate here is, there can be different levels of access granted to different level of roles in an organization to access data.

Dormant Data and Dark Data are mostly confused with each other.

Dark Data

‘Data that is not utilized is Dark Data.’

Dark Data is data that is not in use due to lack of technology and is not a Data Asset. Dormant Data is the data, which is useable, but decision-makers don't use it, or it’s not required for analysis but that doesn't mean you can delete it or purge it, so we keep it in Tape drives, in Hadoop and nowadays Object Storage is also good choice.

Before the introduction of Big Data, the % age of Dark Data was around 90% as it included semi and unstructured datasets and sometimes structured as well. For example, JSON or XML files, Video or Audio files, Logs, Metadata etc., all these datasets were there but technologies were either not there or the affair to analyze those were very expensive. The only reason companies kept this kind of data was for Compliance or Audit purposes. Since the Big Data ecosystem has come onto the surface and hardware has become dead cheap so hosting and processing these datasets has become possible.

As we discussed in topic i.e., Dark Data, that it is utmost important for a CDO (Chief Data Officer) to minimize Dark Data as much as possible, in the organization. In other words, the more the Dark Data, the more the cost without any ROI. Time is not far when Dark Data Management will become the most essential and obvious KPI for CDO (Chief Data Officer).

Now the questions are.

• Is all Dark Data bad? NO.
• Is all Dark Data delete able or purge able? Yes and No.

‘Data Detritus means that data, which is no more required, which is of no use anymore etc.’

• Dark data is something which was and is, not in-use.
• Dormant data is something which is hardly in-use.
• Detritus data is something which was in-use but will not be in-use, anymore.

In other words, Detritus data was once valuable before it became Dormant, as compared to traditional Dark Data (explained separately) which was never in-use due to technologies barriers etc. Detritus data is normally not deleted or purged as it can still be required for Regulatory or Compliance parties.