Knowledge Hub

Data Tokenization is mostly mixed up with Data Masking but there are major maintenance differences.

‘Data Masking with Mappability’

We know how data masking works i.e., it changes data with random values, means to hide original data so if data reaches into wrong hands, one should not be able to identify the individual to whom the record refers.

Data Tokenization also replaces sensitive data with unique system generated value called Token but during the process original data + token along with mapping details are stored in a Token Vault somewhere externally for safe keeping.

As Token replaces original data in actual data store, its mapping details are still safe in Token Vault so actual data is always reversible in contrast with Data Masking where after sensitive data is randomly changed it’s no longer reversible.

A quick example for experienced Data Warehousing folk is Surrogate Key where original business keys are replaced with unique autogenerate values, but original data is not touched. In Data Warehousing Surrogate Key resides along with actual data whereas in Data Tokenization, Surrogate Key resides in Token Vault.

Data Redaction is confused with data masking, neither it doesn’t work like SMD, but it works pretty much like DDM.

We know how data masking works i.e., it changes original data with random values so if data reaches into wrong hands, the identity of relevant person should not compromise.

Data Redaction also blacks out the data ON THE FLY which may or may not even make sense to the users using it.

There are four types of Data Redaction.

1. Random: Data is masked randomly
2. RedExp: Expressions are masked.
3. Partial: Some of the data is masked
4. Full: Everything is masked

We deal with Data Redaction on daily basis 😊. For example, when we do a Credit/ Debit Card transaction, a message comes on our mobiles. You must have noticed, except last few numbers all card numbers are always STARS i.e., ******* so for layperson those stars are Redacted values where partial data is masked.

Data Pipelines is as simple as traveling of data from one point to another but at the same time, the magnitude and importance of this term is as critical as, in the absence of a reliable Data Pipeline, no Decision can be taken on time, and we know ‘Time is Money’.

Data Processing Mechanisms

• Push: When source systems push data to target systems.
• Pull: When target systems pull data from source systems.
• Streaming: When data keeps flowing from source to target systems.

Data Loading Types

• Bulk Loading: This means processing the whole set of data every time from the source. We also call it ‘Data Dump’.
• CDC Loading: This means processing only changed data from the source.

Type of Data Pipeline

• Batch processing means processing data after a certain time duration e.g., start processing after 24 hours or maybe 12 hours. The better hardware is in place the small-batch processing time-duration can be.
• Real-Time processing (event base) means, the moment there is a change in the source system, that piece of data is processed towards the target e.g., the moment a post is LIKED on Facebook or LinkedIn it moves to downstream systems. To support real-time from OLTP, CDC plays a vital role as in real-time, we expect a very small size of data to travel.

Data Transformation is the bread and butter of ETL developers and data engineers.

Data Transformation is the starting point of any KPI, one needs to pull and process data based on business requirements for decision makers.

E = Extract | T = Transform | L = Load

This is the most common interview question and only those who have architecturally worked on integration can answer.

The general definitions

• ETL: Extract, Transform and Load
• ELT: Extract, Load and Transform
• ECL: Extract, Convert and Load

BUT technically HOW? Ask this nerd kind of question with experienced ETL/ ELT developers and I am sure 90% won't be able to respond.

Let’s go a little deeper and understand.

1. ETL processing is done on the source system as 1) Extract part (select) runs on source system, 2) Transform part runs on source system and 3) Load part (push data to- target table) runs on source system. Means all the operations are happening on the source system.
2. ELT processing has three more operations i.e., its DDL-EL-ETL. In ELT 1) first, a temp table is created (DDL) in the target system, 2) Extract part (select) runs on the source system, 3) Load (one-to-one push to temp table) runs on the source system, 4) Again Extract part (select) from temp table runs on the target system, 5) Transform part runs on the target system and 6) Load part (push to the table) runs on the target system. This means, 2 out of 6 operations are run on the source system where 4 operations including the most resource-hungry operation i.e., TRANSFORM operation run in the target system.
3. ECL processing is majorly used in Text Analytics where ECL tools read text from mediums like word, PDF, text file etc., catalog it by giving tags and load it in target storage which can be anything. Please note, here we are not talking about converting unstructured data like video, audio, scan, image etc., into structured data. Here, we are talking about converting or tagging existing data which is already in text format.

Business Analytics has been the most demanding cycle in the current era. Business Analytics supports DSS (Decision Support Systems) and the technologies to support started from OLTP systems to OLAP systems i.e., source systems to Data Warehouses or Data Marts or Data Lakes or Data Lake Houses or Data Vault or Business Vault or Enterprise Data Hubs or Enterprise Data Fabric etc., but that’s not the complete cycle. The cycle completes once Reverse ETL is performed.

Let’s understand with an example. For example, the product sales data is loaded in a system (OLTP) then it is pushed or pulled in a Data Lake or a Data/ Business Vault or in a Data Warehouse or Data Mart or Data Lake House etc. Once data has move in these datastores then analytics are performed using Data Science practice like by running Machine Learning, Deep Learning, and many other means to analyze data for prediction or forecasting and last but not the least the data is present to Top Management for decision marking. Management decides to initiate campaigns based on the data element extracted from the whole cycle from OLTP to OLAP etc.

Data Conversion is mostly confused with Data Transformation.

In Data Conversion, data is not changed whereas in Data Transformation, data can be changed based on business requirements for certain decision making.

• Data Conversion can be used to standardize different formats into one unified format.
• Data Conversion can be, to convert one format into different formats.
• Data Conversion can be, to convert one format to another format as per business requirement.
• Data Conversion can be, to convert one data model to another using Data Transformation process for business ease of use.
• Data Conversion can be, to convert data from data to Reporting and Dashboards formats.

Example of basic Data Conversion

• Convert weather data like from Celsius to Fahrenheit and vice versa.
• Conversion of all date formats in one format.
• Conversion of unstructured like image, pdf, video, audio etc. into semi or structured file formats
• If a field has numbers only and the data type is string, then better to convert it to number as processing is much faster on number data type as compared to string.
• If a field has predefined data like Male/ Female, Yes/ No etc. and its field data type is varchar or varchar 2 then better to convert it to char data type as processing is much faster on char data type as compared to varchar or varchar 2.

Data Parsing and Formatting sometime works interchangeably and mixed with Data Conversion and Data Transformation which is not correct.

Parsing term comes from Latin part (orationis), meaning part (of speech) [Wikipedia].

We all must have heard about parser. Yes, Data Parsing is about rearranging or formatting the data in such a way that it’s understandable or usable. In Data Parsing, data is not anonymized neither does it changes its meaning rather it changes for the sake of its understandable.

Referring to the image, we can see data in columns are rearranged and now it can make sense. If we pull it in a reports or dashboard, the decisions can be taken, of course not on un-parsed data.

In the era of social media, most of online portals gives APIs to allow third parties to extract data in a readable format but those who doesn’t provide APIs, we use Web Scrapping (explained in separate topic) to crawl the websites, online portals etc., to extract details by running different type of parsing approaches.

CDC and Real-Time, a couple of terms which people are confused with and assumes both are same but in the real world, both are different topics.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

There are Four CDC methods:

1. Keep the previous data set in table1, get a new data set in table2. Compare both and extract rows that are deleted or are new or have changed since last time. This is the slowest method and requires the heaviest processing either on the source (not recommended) or on target system.
2. Source Tables have date columns that can identify the latest inserted or updated records. Using these columns, CDC data can be extracted easily. This is the easy approach, but it requires the Source Data Model to have date fields in each table which if not requires extensive development effort both at the backend database and frontend Application. It has a processing impact on the source system as well.
3. This method is mostly used where Transaction Logs are read. Transaction Log is generated when an insert/ update/ delete operation takes place on a table. In this method, the log is read instead of hitting the source tables so no impact on the source database but minimal impact on I/O [recommended and mostly used approach].
4. This is the most recent method and very few solutions are using it. In this method, the data is read at the application level, in other words when the Application writes in Database, at the same time it pushes the operation for CDC processing. No impact on source database or I/O.

ESP (Event-Stream Processing) is a term used along with CDC and Real-Time processing.

As we touched based on Real-Time processing in another topic, ESP is exactly what Real-Time is all about. In other words, the moment any activity has happened in source system or in source application which can be online portal, social media or a mobile etc., so the moment any activity takes place, the latest action is pushed to next stage. Next stage can be a Data Lake or Data Warehouse, or it can be intermediately solution in in-memory to prepare a response to send directly back to customer’s mobile.

‘Even-Streaming Processing is heavily used if a reaction is planned in Real-Time.’

For example, a customer buys a product from a shop in a mall, the moment he or she use their credit card for payment, along with OTP code another message comes to their mobile, telling them there is a sales or promotion going on in this mall, please have a look before making this payment. This is called Real-Time Even-Streaming Processing.

Security and privacy are often mentioned in the same breath, but they serve distinct, and complementary, purposes. Both are essential pillars of responsible data management in the digital era.

Data Security is about protection. It ensures data remains confidential, accurate, and accessible only to those who are authorized. Security safeguards against unauthorized access, malicious attacks, accidental loss, and corruption of information. Techniques include access management, activity monitoring, encryption, masking, breach detection, and disaster recovery. In short, security preserves the confidentiality, integrity, and availability (CIA) of data.

Data Privacy is about rights and responsible use. It governs who can access what data, for what purpose, and under what conditions. Privacy extends beyond technical controls, covering contracts, policies, consent management, and compliance with laws such as the GDPR in Europe, the PDPA in Singapore and Malaysia, or the CCPA in the United States. Privacy ensures that individuals’ personally identifiable information (PII) is collected minimally, used ethically, and safeguarded throughout its lifecycle.