Knowledge Hub

Data Parsing and Formatting sometime works interchangeably and mixed with Data Conversion and Data Transformation which is not correct.

Parsing term comes from Latin part (orationis), meaning part (of speech) [Wikipedia].

We all must have heard about parser. Yes, Data Parsing is about rearranging or formatting the data in such a way that it’s understandable or usable. In Data Parsing, data is not anonymized neither does it changes its meaning rather it changes for the sake of its understandable.

Referring to the image, we can see data in columns are rearranged and now it can make sense. If we pull it in a reports or dashboard, the decisions can be taken, of course not on un-parsed data.

In the era of social media, most of online portals gives APIs to allow third parties to extract data in a readable format but those who doesn’t provide APIs, we use Web Scrapping (explained in separate topic) to crawl the websites, online portals etc., to extract details by running different type of parsing approaches.

CDC and Real-Time, a couple of terms which people are confused with and assumes both are same but in the real world, both are different topics.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

There are Four CDC methods:

1. Keep the previous data set in table1, get a new data set in table2. Compare both and extract rows that are deleted or are new or have changed since last time. This is the slowest method and requires the heaviest processing either on the source (not recommended) or on target system.
2. Source Tables have date columns that can identify the latest inserted or updated records. Using these columns, CDC data can be extracted easily. This is the easy approach, but it requires the Source Data Model to have date fields in each table which if not requires extensive development effort both at the backend database and frontend Application. It has a processing impact on the source system as well.
3. This method is mostly used where Transaction Logs are read. Transaction Log is generated when an insert/ update/ delete operation takes place on a table. In this method, the log is read instead of hitting the source tables so no impact on the source database but minimal impact on I/O [recommended and mostly used approach].
4. This is the most recent method and very few solutions are using it. In this method, the data is read at the application level, in other words when the Application writes in Database, at the same time it pushes the operation for CDC processing. No impact on source database or I/O.

ESP (Event-Stream Processing) is a term used along with CDC and Real-Time processing.

As we touched based on Real-Time processing in another topic, ESP is exactly what Real-Time is all about. In other words, the moment any activity has happened in source system or in source application which can be online portal, social media or a mobile etc., so the moment any activity takes place, the latest action is pushed to next stage. Next stage can be a Data Lake or Data Warehouse, or it can be intermediately solution in in-memory to prepare a response to send directly back to customer’s mobile.

‘Even-Streaming Processing is heavily used if a reaction is planned in Real-Time.’

For example, a customer buys a product from a shop in a mall, the moment he or she use their credit card for payment, along with OTP code another message comes to their mobile, telling them there is a sales or promotion going on in this mall, please have a look before making this payment. This is called Real-Time Even-Streaming Processing.

Security and privacy are often mentioned in the same breath, but they serve distinct, and complementary, purposes. Both are essential pillars of responsible data management in the digital era.

Data Security is about protection. It ensures data remains confidential, accurate, and accessible only to those who are authorized. Security safeguards against unauthorized access, malicious attacks, accidental loss, and corruption of information. Techniques include access management, activity monitoring, encryption, masking, breach detection, and disaster recovery. In short, security preserves the confidentiality, integrity, and availability (CIA) of data.

Data Privacy is about rights and responsible use. It governs who can access what data, for what purpose, and under what conditions. Privacy extends beyond technical controls, covering contracts, policies, consent management, and compliance with laws such as the GDPR in Europe, the PDPA in Singapore and Malaysia, or the CCPA in the United States. Privacy ensures that individuals’ personally identifiable information (PII) is collected minimally, used ethically, and safeguarded throughout its lifecycle.

DLP stands for Data Loss Protection/ Prevention which is part of the organizational level security strategy.

‘DLP is to protect or prevent data from getting in wrong hands.’

DLP is normally handled and maintained by the Risk department in an organization where standards and policies are followed to protect data from intruders. We know, the starting point of all the standards and policies start from Regulatory bodies.

The moment you expose your data to outsider via Internet, the DLP comes into action. One must take care of their own data from intruders and hackers. In previous era we had the choices to connect to external world or not, but now NO, we don’t have a choice, we must connect to social media to pull data, we must connect to cloud etc.

Most people confuse Quality and Data Integrity with each other.

The term Data Integrity came from a security point of view, but it came from sharing data in a more effective and easier way, so in a way, it is more than just security. It has multiple components and Data Quality is one of those. In other words, these are not two different terms, nor both compete, rather Data Quality supplements Data Integrity along with other components.

Achieving a higher % age of Data Integrity gives confidence to the business users and management to trust data for daily decision making. Losing Data Integrity means a higher possibility of wrong data, wrong data means wrong decisions, and wrong decisions mean business lose margin or go into loss. Security’s point of view is to make sure the results shared with Public Stakeholders or Regulatory Bodies are correct and not fake. That's the reason there are internal and external audit teams evaluating the Data Integrity of organizations. Based on audit results, more investors invest, more buyers buy, and the company flourishes.

Data Compliance is mostly confused with Data Quality and Data Integrity, which is somehow true, as all these terms do share some common features for ONE GOAL i.e., Data Protection.

Data Compliance is to follow certain regulations and standards of Data Security, Data Privacy and Data Integrity, set by different Regulatory Bodies around the World to protect sensitive data and to make data available in certain ways.

Regulatory Bodies set, if not all but, CERTAIN Data Security, Data Privacy and Data Integrity standards. So, having only Data Compliance measures will not protect data 100%, there are many other standards covered under Data Security, Data Privacy and Data Integrity which are required to be in-place for any organization.

Common Regulatory Bodies Frameworks

We won’t be going into details of regulatory bodies like what is required by each but for the audience, let me mention the most famous ones here.

Data Preservation, Data Security, Data Privacy, Data Integrity, Data Compliance all refer to the protection of the data.

Data Preservation is about securing and maintaining the safety and integrity of the data after it’s been processed, for future reference.

‘Data which is preserved cannot be changed and hardly purged or deleted.’

Things to consider for Data Preservation

• What data to be preserved? Not all data is worth preserving.
• What is the retention period to preserve data? Like for how long to keep data.
• Who will be responsible for preserving data? A custodian is required for data repository.

In the current era with big data having huge semi and unstructured datasets, it’s not worth rather not possible to preserve everything. One must decide what to preserve and for how long.

For example, in datastore like Data Warehouse or a Data Mart or in Data Lake etc., there are certain datasets to preserve. One can start from the list of compliance requirements e.g., traditionally in banking sector, the customers who has left the bank or in other words closed their accounts, their history is preserved or saved for 7 years from the day they closed their accounts.

A term very near to Data Security and Data Privacy. Data Sovereignty is about keeping the data secure within the defined boundaries, especially e.g., certain sets of data should not go cross border for certain sectors.

In Data Security and Data Privacy, we make sure data is only accessed by those who need it or who own it. Within the organization, we maintain an ACL (Access Control List) to record the list of all the users with their level of access they can have on certain data sets. For example, which level of access developers can have, which level of access database administrators can have, which level of access department’s manager can have, which level of access a Business User can have, which of access Cost Centre person can have etc. So, in a way what we are trying to elaborate here is, there can be different levels of access granted to different level of roles in an organization to access data.

Data Virtualization is the virtual consumption layer and connect with anonymous data stores i.e., RDBMS, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. As it’s connecting directly with all data stores, the data models of all systems remain intact with no change.

‘It’s considered as one of the solutions for Data Sovereignty problem statement.’

What data stores?

Here data stores are referred as traditional RDBMS or OLTP systems, OLAP systems, Data Warehouse, Data Lake, Delta Lake, Data Vault, Business Vault, Kafka, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. Yes, it’s confusing as we all know above data stores serve their purpose for one or the other business requirement then why this another layer on top?

Data Virtualization hosts views connecting directly with all possible data stores. These views can be one-to-one or can be complex SQL written views to pull data from multiple source systems to create a virtualized layer on top of consumer layer to consume.