Knowledge Hub

ESP (Event-Stream Processing) is a term used along with CDC and Real-Time processing.

As we touched based on Real-Time processing in another topic, ESP is exactly what Real-Time is all about. In other words, the moment any activity has happened in source system or in source application which can be online portal, social media or a mobile etc., so the moment any activity takes place, the latest action is pushed to next stage. Next stage can be a Data Lake or Data Warehouse, or it can be intermediately solution in in-memory to prepare a response to send directly back to customer’s mobile.

‘Even-Streaming Processing is heavily used if a reaction is planned in Real-Time.’

For example, a customer buys a product from a shop in a mall, the moment he or she use their credit card for payment, along with OTP code another message comes to their mobile, telling them there is a sales or promotion going on in this mall, please have a look before making this payment. This is called Real-Time Even-Streaming Processing.

Security and privacy are often mentioned in the same breath, but they serve distinct, and complementary, purposes. Both are essential pillars of responsible data management in the digital era.

Data Security is about protection. It ensures data remains confidential, accurate, and accessible only to those who are authorized. Security safeguards against unauthorized access, malicious attacks, accidental loss, and corruption of information. Techniques include access management, activity monitoring, encryption, masking, breach detection, and disaster recovery. In short, security preserves the confidentiality, integrity, and availability (CIA) of data.

Data Privacy is about rights and responsible use. It governs who can access what data, for what purpose, and under what conditions. Privacy extends beyond technical controls, covering contracts, policies, consent management, and compliance with laws such as the GDPR in Europe, the PDPA in Singapore and Malaysia, or the CCPA in the United States. Privacy ensures that individuals’ personally identifiable information (PII) is collected minimally, used ethically, and safeguarded throughout its lifecycle.

DLP stands for Data Loss Protection/ Prevention which is part of the organizational level security strategy.

‘DLP is to protect or prevent data from getting in wrong hands.’

DLP is normally handled and maintained by the Risk department in an organization where standards and policies are followed to protect data from intruders. We know, the starting point of all the standards and policies start from Regulatory bodies.

The moment you expose your data to outsider via Internet, the DLP comes into action. One must take care of their own data from intruders and hackers. In previous era we had the choices to connect to external world or not, but now NO, we don’t have a choice, we must connect to social media to pull data, we must connect to cloud etc.

Most people confuse Quality and Data Integrity with each other.

The term Data Integrity came from a security point of view, but it came from sharing data in a more effective and easier way, so in a way, it is more than just security. It has multiple components and Data Quality is one of those. In other words, these are not two different terms, nor both compete, rather Data Quality supplements Data Integrity along with other components.

Achieving a higher % age of Data Integrity gives confidence to the business users and management to trust data for daily decision making. Losing Data Integrity means a higher possibility of wrong data, wrong data means wrong decisions, and wrong decisions mean business lose margin or go into loss. Security’s point of view is to make sure the results shared with Public Stakeholders or Regulatory Bodies are correct and not fake. That's the reason there are internal and external audit teams evaluating the Data Integrity of organizations. Based on audit results, more investors invest, more buyers buy, and the company flourishes.

Data Compliance is mostly confused with Data Quality and Data Integrity, which is somehow true, as all these terms do share some common features for ONE GOAL i.e., Data Protection.

Data Compliance is to follow certain regulations and standards of Data Security, Data Privacy and Data Integrity, set by different Regulatory Bodies around the World to protect sensitive data and to make data available in certain ways.

Regulatory Bodies set, if not all but, CERTAIN Data Security, Data Privacy and Data Integrity standards. So, having only Data Compliance measures will not protect data 100%, there are many other standards covered under Data Security, Data Privacy and Data Integrity which are required to be in-place for any organization.

Common Regulatory Bodies Frameworks

We won’t be going into details of regulatory bodies like what is required by each but for the audience, let me mention the most famous ones here.

Data Preservation, Data Security, Data Privacy, Data Integrity, Data Compliance all refer to the protection of the data.

Data Preservation is about securing and maintaining the safety and integrity of the data after it’s been processed, for future reference.

‘Data which is preserved cannot be changed and hardly purged or deleted.’

Things to consider for Data Preservation

• What data to be preserved? Not all data is worth preserving.
• What is the retention period to preserve data? Like for how long to keep data.
• Who will be responsible for preserving data? A custodian is required for data repository.

In the current era with big data having huge semi and unstructured datasets, it’s not worth rather not possible to preserve everything. One must decide what to preserve and for how long.

For example, in datastore like Data Warehouse or a Data Mart or in Data Lake etc., there are certain datasets to preserve. One can start from the list of compliance requirements e.g., traditionally in banking sector, the customers who has left the bank or in other words closed their accounts, their history is preserved or saved for 7 years from the day they closed their accounts.

A term very near to Data Security and Data Privacy. Data Sovereignty is about keeping the data secure within the defined boundaries, especially e.g., certain sets of data should not go cross border for certain sectors.

In Data Security and Data Privacy, we make sure data is only accessed by those who need it or who own it. Within the organization, we maintain an ACL (Access Control List) to record the list of all the users with their level of access they can have on certain data sets. For example, which level of access developers can have, which level of access database administrators can have, which level of access department’s manager can have, which level of access a Business User can have, which of access Cost Centre person can have etc. So, in a way what we are trying to elaborate here is, there can be different levels of access granted to different level of roles in an organization to access data.

Data Virtualization is the virtual consumption layer and connect with anonymous data stores i.e., RDBMS, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. As it’s connecting directly with all data stores, the data models of all systems remain intact with no change.

‘It’s considered as one of the solutions for Data Sovereignty problem statement.’

What data stores?

Here data stores are referred as traditional RDBMS or OLTP systems, OLAP systems, Data Warehouse, Data Lake, Delta Lake, Data Vault, Business Vault, Kafka, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. Yes, it’s confusing as we all know above data stores serve their purpose for one or the other business requirement then why this another layer on top?

Data Virtualization hosts views connecting directly with all possible data stores. These views can be one-to-one or can be complex SQL written views to pull data from multiple source systems to create a virtualized layer on top of consumer layer to consume.

Data Federation works just like Data Virtualization framework but what is the difference between both.

Relationship between Data Federation and Data Virtualization is just like relationship between Data Mining and Data Science (there is separate topic on Data Mining vs Data Science).

Conceptually, both Data Federation and Data Virtualization works the same way, but Data Federation is restricted to relational database whereas Data Virtualization also includes data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc., as well.

Another major difference i.e., Data Federation doesn’t create a unified data model whereas Data Virtualization does.

Below are a few similar behaviors of both Data Federation and Data Virtualization.

• Single Access Point: Access all data stores from a single environment.
• UAM: Use Access Management at one place.
• Data Quality Management: Data Quality issues can be identified from a single point.
• No Data Movement: No ETL/ ELT is required.
• Ease of use: As users don’t have to write queries in different languages. Users can write in one language, and it will create a wrapper to talk to all the connected systems.
• No Additional Storage: As a virtual copy there is no cost for any separate copy of data, so no additional storage is required.
• Low Risk: As it’s not hosting the data copy as compared to Data Warehouses or Data Lake where data is physically moved, there is no chance of missing any data.
• Better for Analytics: As all the source data models are connected and data is fetched in real like, it makes source data available in real-time for Data Analysts and Data Scientist
• Performance Impact: Some impact on source systems as some queries will run at source.
• Functional Knowledge: Seek dedicated involvement by Data Stewards and Functional consultants.
• No Historical Data: As there is no copy of data hosted so history data is also not maintained. These are best suited along with Data Warehouse in-place.

Data Virtualization, Data Federation and Data Consolidation all are discussed together with respect to exposing data to data consumers.

Where Data Virtualization and Data Federation do not host any copy of data in its environments, Data Consolidation on the other hand keeps complete historical copies of all the data coming from all the source systems. In this process sometimes, a complex ETL or ELT takes place to process data in different data models and sometimes one to one data model is maintained.

Yes, I am referring to Data Warehouses, Data Marts, Data Lakes, Delta Lakes, Data Vault, Business Vaults etc., all these data stores fall under the umbrella of Data Consolidation environments.