Knowledge Hub

Data Lifecycle Management (DLM) is mostly confused with Data Supply Chain which is not 100% correct.

Data Lifecycle Management is about the process from where data was generated till it was removed from the systems whereas Data Supply Chain is about where data come from and where it goes.

There are many sets of steps for Data Lifecycle management, we will be looking into the most commonly used.

Steps in Data Lifecycle Management (DLM)

• Data Generation means the channel where data is getting generated e.g., mobile application, CRM, ERP, HRM, CCTV, Social-Media, Audio(s), Emails, Word document, PDF, Images, Scans etc.
• Data Processing means, what are the business rules based on which raw data is transformed in required format which can be via ETL or ELT or via Data Mining or Data Science Machine Learning Modelling etc.
• Data Reconciliation, Presentation and Utilization means how transformed data is matched with raw data then how it’s been shown on Reports and Dashboard, and lastly for what purpose it has been used.
• Data Storage Lifespan and Extraction means, for how long the data will be kept and what are the means to extract it which can be manual or can be via automated tools.
• Data Removal means how data has been removed and based on which criteria(s).

Data Usability is the key to being a Data Driven Organization.

To better understand, let’s refer to the difference between Useful and Usable.

• Useful means e.g., a system is fulfilling the business requirements and able to perform all the operation tasks.
• Usable means e.g., how easy the system is for operational tasks i.e., how automated it is, which makes the operator’s life easy.

Let’s map it with Data World. Consider, the marketing team needs data from, let’s say, 5 source systems to send campaigns to premium bank customers. Now, dumping data from all 5 source systems at one centralized location like Data Lake might be USEFUL as all sort of data is available at one place but it’s not USABLE unless until it is not converted into a format which marketing team can easily use to send campaigns. In other words, unless data is not formatted in a Data Warehouse or a Data Mart or in a Semantic Layer, it might not be usable for relevant team.

As the name says all, Data Sharing is about making data accessible to all the relevant stakeholders.

‘Data Sharing is about eliminating Dark Data.’

Data Sharing is one of the key components in Data Driven Organization where organizations aim to build data culture with Data Liberalization.

Data Sharing Channels

• Using Business Intelligence tools
• Using languages like SQL, Python etc.
• Using APIs or direction Using JDBC/ ODBC Connectors
• Using built-in Drivers
• Using Machine Learning or Using Data Mining Models
• Exporting in Excel, CSV, FF formats
• Pushing data to other databases and applications
• Letting other databases and application to Pull data.
• ETL/ ELT tools
• Audit, Compliance or Regulatory Reports etc.

Data Aggregation is the most used term in our daily lives whether it’s in office or in home, we use it everywhere.

‘Data Aggregation is about converting details into summarized format.’

For example, if someone asks you how many months there are in a year? You won’t be taking names of all the months rather you will answer, there are 12 months.

Referring to the image, if your manager asks how many installations our team has done, you rather sending left side details, you will send right side summarized report to your manager i.e., in-total there are 6 iOS and 3 Android installations.

As we normally use databases or data stores to store data, there are different aggregation functions normally used to summarize the data.

Aggregation Functions as per ISO Standards

• COUNT
• SUM
• AVG
• MIN
• MAX

Few other Aggregation Functions

• APPROX_COUNT_DISTINCT
• CHECKSUM_AGG
• COUNT_BIG
• GROUPING
• GROUPING_ID
• STDEV
• STDEVP
• STRING_AGG
• VAR, VARP

Data Profiling is used by Data Modelers, Data Engineers, Data Governance and Data Quality team as their daily routine.

Data Profile is done using many methods to summarize data for analysis. The aim is to find out the data patterns.

For example, if you want to find out, is there a person in your customer list whose age is above 90 years? Now, one way is to go through each line of data and find out who is aged more than 90 years, or the other way is to run a DINTINCT command on age column which will give you all-unique ages from age column. Consider, if there are customers from age 21 to 100 in your database then unique values should not be more than 80.

Data Profiling is also heavily used by Data Quality teams to find out whether data is following standards defined by Data Quality teams. For example, there is Data Quality Rule i.e., all date columns in the system must follow date format as DD/MM/YYYY. Now, as data is flowing-in on daily basis and sometimes in real-time manner, Data Quality teams have automated Data Profiling functions running to run DISTINCT commands to see if there is any other date format coming in the data. Based on the above date format example, there should always be one unique row from date fields.

Data Standardization can be imposed at a data level or at a data store level.

Data Standardization means converting data into such a format that it’s accessible, useable, readable, and actionable.

Data Standardization reduces manual efforts, increase efficiency, enable segmentations, remove duplications, increase data quality, enforce data governance, increase business opportunities, improve analysis and analytics.

Steps for Data Standardization

• Know your data and what it is required for.
• Recognize data entry points.
• Outline the data standards.
• Standardize your data with the help of data scoring.

Data Standardization Approaches

• Converting all dates format into single format.
• Arranging or sorting database on requirements.
• Converting unstructured data set into structured data set e.g., video, audio, image, pdf, email, word etc. in one readable format.
• Converting multiple semi-structured formats in one format e.g., JSON and XML in one readable format.
• Having Gold Records using MDM system.
• Via Data Profiling process
• Via Data Transformation process
• Via Data Virtualization environment
• Data pipeline processing at one given time i.e., real-time, daily, monthly etc.
• Exposing data in the shape of Reports and Dashboards.
• Converting all data from different language to one language e.g., English.
• Bring data from all relevant source systems into one Unified Data Model in a Data Warehouse or in Star or Snowflake Schema or in Semantic Layer or having a multi-dimensional Cube.
• Enrichment of metadata of the data and Creation "Quality gates" that indicate freshness and anomaly.

Geocoding has become of the most demanding data element in current era where data is generated everywhere and by everyone.

‘Geocoding refers to the geographic location.’

Geocoding is about the name, street, city, regions etc., rather exact location on Earth. The exact data elements representing Geocoding are Longitude and Latitude, also known as Lat and Long. Using Lat and Long, one can reach the exact location, anywhere in the World.

In the current era social media, mobile applications, sensors, satellite scans, CCTV, online video(s), audio(s) etc., all have become a norm. Millions of users are using mobile on a daily basis, everywhere, all the time around the globe. Location information has become one of the most important pieces of information for everyone especially consumer details.

For example, we all know Amazon, it’s one the most used online stores around the globe which is used by millions of people to buy and sell products, 24/7. If Amazon does not know where the buyer is from, and what is the location where most buyers of a particular product are, then it can’t plan its marketing campaigns.

Data Matching and Linking is for multiple purposes across systems.

Data Matching and Linking is about matching and linking datasets across the systems to find out which is the best record to link with. Generating Golden Record for MDM (explained separately) can be one of the use cases. Additionally, we are also referring to when the same nature of data is coming from multiple source systems and there is a need to find the best possible record.

Please note, we are not talking about manual matching and linking, we are talking about complex software(s) using complex algorithms to find the right data.

For example, customer onboard process can be done Online or Customer Service Centre or at a Branch. If the same customer gets registered 2 to 3 times via different channels, then one must use Data Matching and Linking process to identify customer with high weightage.

Types

• Referential is where exact fields are matched and linked like National ID, Social Security Number, Password numbers etc.
• Probabilistic is where multiple fields are used like name, date of birth, parent’s names etc.
• ML Association is where machine learning searches for matching data element to link.

Data Serving is also known as the Serving Layer and it has been there since its inception but as technologies have matured, many new channels have been introduced to support this layer.

‘Serving Layer is last component of the Lambda Architecture.’

Lambda Architecture has been explained in separate topic, so we won’t go into details of it but referring to it, as mentioned above, it is the last layer which is exposed to End Users for reporting purposes. This Layer is, if not all the time but majorly designed based on business requirements.

We all know OLTP systems are not meant to be exposed for reporting purposes. In the current era, for reporting, there are many data stores available i.e., Data Warehouse, OLAP, Data Lake, NoSQL, Semantic layers but ideally Semantic Layer should be treated as Serving Layer.

To be more specific

• If Serving Layer is designed for specific business requirement then Semantic Layer is last layer.
• If the Serving Layer is designed for generic use then it can be Data Warehouse, OLAP, Data Lake, NoSQL etc.

Consumption layer is confused with Serving Layer due to its name. Folks relates it that the layer which is used by Consumers to extract data. The Layer which Consumers use to extract information is Serving layer (explained separately).

Consumption Layer, as its name says it all, is a layer which consumes or takes-in data from source systems. In the current era, due to huge amounts of data coming in every second from sources like social media, CCTV, images, audios, sensors data etc. technology was required to consume data on a fast pace in MPP mode.

Tools like Hadoop, Object Storage and many RDBMS are giving MPP features now a days.

In Lambda Architecture, Serving Layer is the last layer, Consumption Layer is the layer which process data from source till Serving layer.

In the image, Batch and Speed Layers are Consumption Layers. In an OLAP system, the consumption layer is Data Warehouse and Data Mart, considering there are OLAP Cubes and Semantic layers but in a Data Warehouse, consumption layer is Landing and Staging zone.