Knowledge Hub

Data Inventory is a very simple term but sometime get a little bit confused with Data Catalog and Data Assets etc.

Data Assets is about knowing which kind of data we have in our organization with monetary value.

Data Catalog is about knowing system details, databases, tables, and columns and then mapping those with business requirements.

Data Inventory is about consolidating information from Data Asset and Data Catalog with addition details like.

• where this data comes from
• who generated this data?
• when it was generated
• frequency of inserts, updates, and deletes.
• contents details
• security and privacy approaches
• metadata management approaches
• data quality standards
• transformation details
• business rules
• data models
• list of software
• name of all the applications
• licensing details
• data sources details
• target systems details.
• owners of source and target systems

We all maintain inventory of something or the other, either it’s at home or office. A simple example is Passwords Inventory, now a days there are many social media applications then Hotmail, Gmail, etc. We normally maintain all those passwords somewhere safe right so that is your Password Inventory.

Data Curation is about generating, managing, and maintaining data for future use.

‘Curation comes from the Latin word curae, meaning "care for" -- a meaning it still has today, including in relation to data.’

Data Curation is about taking care of the data from start till end. From extracting till loading. From management to governance. From raw till transformed. From generation till secured and private storage. From open till encryption. As the name says, CARE FOR. Data Curation is also about taking steps to preserve data for future reference as well. Data Curation is the process of preparing data for business and for decision support systems. There is nothing complex about its definition 😊.

Steps for Data Curation

• Understanding the data
• Transformation
• Metadata Management
• Cleansing
• Reconciliation

Stages of Data Curation

• Storing --> Sharing --> Discoverable

Syndication is also known as synchronization, yes now it must be making sense to you 😊.

‘Data Syndication is a process not a technology.’

The word syndicate comes from the French word syndicate which means "trade union" (syndic meaning "administrator"), from the Latin word syndicus which in turn comes from the Greek word σύνδικος (syndikos), which means "caretaker of an issue"; compare to ombudsman or representative.

‘Data Syndication is to collect and prepare update-to-date dataset for a particular activity.’

For example, prepare a dataset from multiple source systems for the marketing team to plan their upcoming campaigning exercise.

Data Syndication process

• Can handle any type of format as input and output as well.
• It can send alerts.
• It can configure access matrix.
• It provides monitoring dashboards and reports for compliance and audit trials.
• As it’s a process, any expensive or cheap technology can be utilized.
• Output formats can be designed based on customer requirements.
• Can send email, SMS, spreadsheet, database, ML models etc.

Data Supply Chain is not a data capability rather a process.

Just like generic supply chain, Data Supply Chain works in the same manner where it starts from Source Data Stage 🡪 Transformation and Integration Stage 🡪 Consumption Stage

• Source Data Stage is the first stage where raw data is planned and identified i.e., OLTP systems.
• Transformation and Integration Stage is where data is converted based on business requirement in-shape of Data Lake Systems or Data Warehouses
• Consumption Stage is where data is store for business use in-shape of Data Lake Systems or Data Warehouses (OLAP systems) or Semantic Layer etc.

Data Supply Chain is a very critical operational process to manage Data Asset and Liabilities. In the absence of visibility on Data Supply Chain, organization cannot plan ROI as well as TTM which is critical for the Data Monetization.

Note: Data Assets, Data Liabilities and Data Monetization all are explained in separate topics.

Data Lifecycle Management (DLM) is mostly confused with Data Supply Chain which is not 100% correct.

Data Lifecycle Management is about the process from where data was generated till it was removed from the systems whereas Data Supply Chain is about where data come from and where it goes.

There are many sets of steps for Data Lifecycle management, we will be looking into the most commonly used.

Steps in Data Lifecycle Management (DLM)

• Data Generation means the channel where data is getting generated e.g., mobile application, CRM, ERP, HRM, CCTV, Social-Media, Audio(s), Emails, Word document, PDF, Images, Scans etc.
• Data Processing means, what are the business rules based on which raw data is transformed in required format which can be via ETL or ELT or via Data Mining or Data Science Machine Learning Modelling etc.
• Data Reconciliation, Presentation and Utilization means how transformed data is matched with raw data then how it’s been shown on Reports and Dashboard, and lastly for what purpose it has been used.
• Data Storage Lifespan and Extraction means, for how long the data will be kept and what are the means to extract it which can be manual or can be via automated tools.
• Data Removal means how data has been removed and based on which criteria(s).

Data Usability is the key to being a Data Driven Organization.

To better understand, let’s refer to the difference between Useful and Usable.

• Useful means e.g., a system is fulfilling the business requirements and able to perform all the operation tasks.
• Usable means e.g., how easy the system is for operational tasks i.e., how automated it is, which makes the operator’s life easy.

Let’s map it with Data World. Consider, the marketing team needs data from, let’s say, 5 source systems to send campaigns to premium bank customers. Now, dumping data from all 5 source systems at one centralized location like Data Lake might be USEFUL as all sort of data is available at one place but it’s not USABLE unless until it is not converted into a format which marketing team can easily use to send campaigns. In other words, unless data is not formatted in a Data Warehouse or a Data Mart or in a Semantic Layer, it might not be usable for relevant team.

As the name says all, Data Sharing is about making data accessible to all the relevant stakeholders.

‘Data Sharing is about eliminating Dark Data.’

Data Sharing is one of the key components in Data Driven Organization where organizations aim to build data culture with Data Liberalization.

Data Sharing Channels

• Using Business Intelligence tools
• Using languages like SQL, Python etc.
• Using APIs or direction Using JDBC/ ODBC Connectors
• Using built-in Drivers
• Using Machine Learning or Using Data Mining Models
• Exporting in Excel, CSV, FF formats
• Pushing data to other databases and applications
• Letting other databases and application to Pull data.
• ETL/ ELT tools
• Audit, Compliance or Regulatory Reports etc.

Data Aggregation is the most used term in our daily lives whether it’s in office or in home, we use it everywhere.

‘Data Aggregation is about converting details into summarized format.’

For example, if someone asks you how many months there are in a year? You won’t be taking names of all the months rather you will answer, there are 12 months.

Referring to the image, if your manager asks how many installations our team has done, you rather sending left side details, you will send right side summarized report to your manager i.e., in-total there are 6 iOS and 3 Android installations.

As we normally use databases or data stores to store data, there are different aggregation functions normally used to summarize the data.

Aggregation Functions as per ISO Standards

• COUNT
• SUM
• AVG
• MIN
• MAX

Few other Aggregation Functions

• APPROX_COUNT_DISTINCT
• CHECKSUM_AGG
• COUNT_BIG
• GROUPING
• GROUPING_ID
• STDEV
• STDEVP
• STRING_AGG
• VAR, VARP

Data Profiling is used by Data Modelers, Data Engineers, Data Governance and Data Quality team as their daily routine.

Data Profile is done using many methods to summarize data for analysis. The aim is to find out the data patterns.

For example, if you want to find out, is there a person in your customer list whose age is above 90 years? Now, one way is to go through each line of data and find out who is aged more than 90 years, or the other way is to run a DINTINCT command on age column which will give you all-unique ages from age column. Consider, if there are customers from age 21 to 100 in your database then unique values should not be more than 80.

Data Profiling is also heavily used by Data Quality teams to find out whether data is following standards defined by Data Quality teams. For example, there is Data Quality Rule i.e., all date columns in the system must follow date format as DD/MM/YYYY. Now, as data is flowing-in on daily basis and sometimes in real-time manner, Data Quality teams have automated Data Profiling functions running to run DISTINCT commands to see if there is any other date format coming in the data. Based on the above date format example, there should always be one unique row from date fields.

Data Standardization can be imposed at a data level or at a data store level.

Data Standardization means converting data into such a format that it’s accessible, useable, readable, and actionable.

Data Standardization reduces manual efforts, increase efficiency, enable segmentations, remove duplications, increase data quality, enforce data governance, increase business opportunities, improve analysis and analytics.

Steps for Data Standardization

• Know your data and what it is required for.
• Recognize data entry points.
• Outline the data standards.
• Standardize your data with the help of data scoring.

Data Standardization Approaches

• Converting all dates format into single format.
• Arranging or sorting database on requirements.
• Converting unstructured data set into structured data set e.g., video, audio, image, pdf, email, word etc. in one readable format.
• Converting multiple semi-structured formats in one format e.g., JSON and XML in one readable format.
• Having Gold Records using MDM system.
• Via Data Profiling process
• Via Data Transformation process
• Via Data Virtualization environment
• Data pipeline processing at one given time i.e., real-time, daily, monthly etc.
• Exposing data in the shape of Reports and Dashboards.
• Converting all data from different language to one language e.g., English.
• Bring data from all relevant source systems into one Unified Data Model in a Data Warehouse or in Star or Snowflake Schema or in Semantic Layer or having a multi-dimensional Cube.
• Enrichment of metadata of the data and Creation "Quality gates" that indicate freshness and anomaly.