Knowledge Hub

Data Generalization/ Blurring is normally confused with Data Anonymization which is about removing or modifying personal information so the identity of a person can be undetectable. Data Generalization aim is same but using with different approach.

‘Data Generalization/ Blurring and Specialization is something like Zoom-Out vs Zoom-In or Precise vs Less-Precise.’

One of the main requirements for Data Generalization comes into play when you must maintain Data Sovereignty (explain in a separate topic) but keep the data originality. It is used to cater for data privacy.

‘Data Generalization summarized data in such a way that the usefulness of the data remains intact.’

Data Specialization is the right opposite of Data Generalization as it divides entity into multiple entities to further make sense of the data.

Example of Data Generalization: Referring to the image, Students and Patients both have Name and Mobile numbers, we can have summarized entity like Person which can host both Students and Patients details i.e., Roll_No and PID numbers. This way, there will be records of all the students and patients, but no one will be able to identify any student or patient.

Data Perturbation/ Swapping/ Shuffling/ Data Scrambling/ Data Obfuscation all use the same technique to hide the identity of an entity but keeping the essence of the data.

Data Swapping is about replacing the values of data at the same level of detail as compared to original data. In other words, cardinality (explained in a separate topic) of the replaced data is maintained.

• In Data Anonymization, identifiable data was removed.
• In Data Generalization, identifiable data is summarized.
• In Data Swapping, the level of identifiable data is replaced.

‘Data Swapping technique is majorly used by Data Scientists or Data Citizen who require data at the lowest level to create and train their Machine Learning Models.’

Let’s refer to the same Age example used in topic: Data Generalization/ Blurring and Specialization. For example, there can be an Age column with value: 26, 28, 31, 33, 37, 42, 42, 46, 48, 49, 54, 57, 57, 58, 59. In Data Swapping, to maintain the same level of detail i.e., ages can be swapped 26 with 59, 28 with 58, 32 with 57 and so on.

Static Data Masking (SDM) is also known as Data Masking and is one of the most underrated terms in Data but at the same time it's one of the most wanted terms for Data Privacy.

Data Masking is from the same family i.e., Data Perturbation/ Swapping/ Shuffling/ Data Scrambling/ Data Obfuscation.

Data Masking is a technique to convert actual data into dummy data for the sake of hiding its originality. If the Credit Card number is 3345-0987-2234-2342, it will be converted into 1234-5678-9101-1123. So, if someone can view my data, my identity will not be disclosed. I am sure understanding Data Masking is not difficult but the implications of not having or having bad techniques are debatable.

Please note, there are many domains where Regulatory bodies don't allow organizations to disclose certain sets of data to anyone which creates challenges in the development environment or at the time of sharing the data to third/ external parties.

Dynamic Data Masking (DDM) behaves just like SDM (Static Data Masking) or in other words like standard Data Masking.

We all know how Data Masking works so we won’t be discussing it, rather we will discuss why DDM came into being and why it’s getting very popular.

As we all know previously, we were working on structured data only which is only 20% whereas in new era, we also have 80% more data i.e., semi and unstructured datasets. Not only datasets, but the data science practices have also matured in the last 10 years due to which more and more Production ready data is required by Data Scientists to develop, train, and publish their Data Science Models.

With the introduction of 80% more data i.e., semi and unstructured datasets, it became more troublesome to anonymize data within given SLA. Static Data Masking is a very slow process where one must create a new dataset with masked data, but using Dynamic Data Masking, Production data is masked on the fly before sharing with relevant parties.

Data Tokenization is mostly mixed up with Data Masking but there are major maintenance differences.

‘Data Masking with Mappability’

We know how data masking works i.e., it changes data with random values, means to hide original data so if data reaches into wrong hands, one should not be able to identify the individual to whom the record refers.

Data Tokenization also replaces sensitive data with unique system generated value called Token but during the process original data + token along with mapping details are stored in a Token Vault somewhere externally for safe keeping.

As Token replaces original data in actual data store, its mapping details are still safe in Token Vault so actual data is always reversible in contrast with Data Masking where after sensitive data is randomly changed it’s no longer reversible.

A quick example for experienced Data Warehousing folk is Surrogate Key where original business keys are replaced with unique autogenerate values, but original data is not touched. In Data Warehousing Surrogate Key resides along with actual data whereas in Data Tokenization, Surrogate Key resides in Token Vault.

Data Pipelines is as simple as traveling of data from one point to another but at the same time, the magnitude and importance of this term is as critical as, in the absence of a reliable Data Pipeline, no Decision can be taken on time, and we know ‘Time is Money’.

Data Processing Mechanisms

• Push: When source systems push data to target systems.
• Pull: When target systems pull data from source systems.
• Streaming: When data keeps flowing from source to target systems.

Data Loading Types

• Bulk Loading: This means processing the whole set of data every time from the source. We also call it ‘Data Dump’.
• CDC Loading: This means processing only changed data from the source.

Type of Data Pipeline

• Batch processing means processing data after a certain time duration e.g., start processing after 24 hours or maybe 12 hours. The better hardware is in place the small-batch processing time-duration can be.
• Real-Time processing (event base) means, the moment there is a change in the source system, that piece of data is processed towards the target e.g., the moment a post is LIKED on Facebook or LinkedIn it moves to downstream systems. To support real-time from OLTP, CDC plays a vital role as in real-time, we expect a very small size of data to travel.

Data Transformation is the bread and butter of ETL developers and data engineers.

Data Transformation is the starting point of any KPI, one needs to pull and process data based on business requirements for decision makers.

E = Extract | T = Transform | L = Load

This is the most common interview question and only those who have architecturally worked on integration can answer.

The general definitions

• ETL: Extract, Transform and Load
• ELT: Extract, Load and Transform
• ECL: Extract, Convert and Load

BUT technically HOW? Ask this nerd kind of question with experienced ETL/ ELT developers and I am sure 90% won't be able to respond.

Let’s go a little deeper and understand.

1. ETL processing is done on the source system as 1) Extract part (select) runs on source system, 2) Transform part runs on source system and 3) Load part (push data to- target table) runs on source system. Means all the operations are happening on the source system.
2. ELT processing has three more operations i.e., its DDL-EL-ETL. In ELT 1) first, a temp table is created (DDL) in the target system, 2) Extract part (select) runs on the source system, 3) Load (one-to-one push to temp table) runs on the source system, 4) Again Extract part (select) from temp table runs on the target system, 5) Transform part runs on the target system and 6) Load part (push to the table) runs on the target system. This means, 2 out of 6 operations are run on the source system where 4 operations including the most resource-hungry operation i.e., TRANSFORM operation run in the target system.
3. ECL processing is majorly used in Text Analytics where ECL tools read text from mediums like word, PDF, text file etc., catalog it by giving tags and load it in target storage which can be anything. Please note, here we are not talking about converting unstructured data like video, audio, scan, image etc., into structured data. Here, we are talking about converting or tagging existing data which is already in text format.

Business Analytics has been the most demanding cycle in the current era. Business Analytics supports DSS (Decision Support Systems) and the technologies to support started from OLTP systems to OLAP systems i.e., source systems to Data Warehouses or Data Marts or Data Lakes or Data Lake Houses or Data Vault or Business Vault or Enterprise Data Hubs or Enterprise Data Fabric etc., but that’s not the complete cycle. The cycle completes once Reverse ETL is performed.

Let’s understand with an example. For example, the product sales data is loaded in a system (OLTP) then it is pushed or pulled in a Data Lake or a Data/ Business Vault or in a Data Warehouse or Data Mart or Data Lake House etc. Once data has move in these datastores then analytics are performed using Data Science practice like by running Machine Learning, Deep Learning, and many other means to analyze data for prediction or forecasting and last but not the least the data is present to Top Management for decision marking. Management decides to initiate campaigns based on the data element extracted from the whole cycle from OLTP to OLAP etc.

Data Conversion is mostly confused with Data Transformation.

In Data Conversion, data is not changed whereas in Data Transformation, data can be changed based on business requirements for certain decision making.

• Data Conversion can be used to standardize different formats into one unified format.
• Data Conversion can be, to convert one format into different formats.
• Data Conversion can be, to convert one format to another format as per business requirement.
• Data Conversion can be, to convert one data model to another using Data Transformation process for business ease of use.
• Data Conversion can be, to convert data from data to Reporting and Dashboards formats.

Example of basic Data Conversion

• Convert weather data like from Celsius to Fahrenheit and vice versa.
• Conversion of all date formats in one format.
• Conversion of unstructured like image, pdf, video, audio etc. into semi or structured file formats
• If a field has numbers only and the data type is string, then better to convert it to number as processing is much faster on number data type as compared to string.
• If a field has predefined data like Male/ Female, Yes/ No etc. and its field data type is varchar or varchar 2 then better to convert it to char data type as processing is much faster on char data type as compared to varchar or varchar 2.

Data Parsing and Formatting sometime works interchangeably and mixed with Data Conversion and Data Transformation which is not correct.

Parsing term comes from Latin part (orationis), meaning part (of speech) [Wikipedia].

We all must have heard about parser. Yes, Data Parsing is about rearranging or formatting the data in such a way that it’s understandable or usable. In Data Parsing, data is not anonymized neither does it changes its meaning rather it changes for the sake of its understandable.

Referring to the image, we can see data in columns are rearranged and now it can make sense. If we pull it in a reports or dashboard, the decisions can be taken, of course not on un-parsed data.

In the era of social media, most of online portals gives APIs to allow third parties to extract data in a readable format but those who doesn’t provide APIs, we use Web Scrapping (explained in separate topic) to crawl the websites, online portals etc., to extract details by running different type of parsing approaches.