Knowledge Hub

Data Conversion is mostly confused with Data Transformation.

In Data Conversion, data is not changed whereas in Data Transformation, data can be changed based on business requirements for certain decision making.

• Data Conversion can be used to standardize different formats into one unified format.
• Data Conversion can be, to convert one format into different formats.
• Data Conversion can be, to convert one format to another format as per business requirement.
• Data Conversion can be, to convert one data model to another using Data Transformation process for business ease of use.
• Data Conversion can be, to convert data from data to Reporting and Dashboards formats.

Example of basic Data Conversion

• Convert weather data like from Celsius to Fahrenheit and vice versa.
• Conversion of all date formats in one format.
• Conversion of unstructured like image, pdf, video, audio etc. into semi or structured file formats
• If a field has numbers only and the data type is string, then better to convert it to number as processing is much faster on number data type as compared to string.
• If a field has predefined data like Male/ Female, Yes/ No etc. and its field data type is varchar or varchar 2 then better to convert it to char data type as processing is much faster on char data type as compared to varchar or varchar 2.

Data Parsing and Formatting sometime works interchangeably and mixed with Data Conversion and Data Transformation which is not correct.

Parsing term comes from Latin part (orationis), meaning part (of speech) [Wikipedia].

We all must have heard about parser. Yes, Data Parsing is about rearranging or formatting the data in such a way that it’s understandable or usable. In Data Parsing, data is not anonymized neither does it changes its meaning rather it changes for the sake of its understandable.

Referring to the image, we can see data in columns are rearranged and now it can make sense. If we pull it in a reports or dashboard, the decisions can be taken, of course not on un-parsed data.

In the era of social media, most of online portals gives APIs to allow third parties to extract data in a readable format but those who doesn’t provide APIs, we use Web Scrapping (explained in separate topic) to crawl the websites, online portals etc., to extract details by running different type of parsing approaches.

CDC and Real-Time, a couple of terms which people are confused with and assumes both are same but in the real world, both are different topics.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

There are Four CDC methods:

1. Keep the previous data set in table1, get a new data set in table2. Compare both and extract rows that are deleted or are new or have changed since last time. This is the slowest method and requires the heaviest processing either on the source (not recommended) or on target system.
2. Source Tables have date columns that can identify the latest inserted or updated records. Using these columns, CDC data can be extracted easily. This is the easy approach, but it requires the Source Data Model to have date fields in each table which if not requires extensive development effort both at the backend database and frontend Application. It has a processing impact on the source system as well.
3. This method is mostly used where Transaction Logs are read. Transaction Log is generated when an insert/ update/ delete operation takes place on a table. In this method, the log is read instead of hitting the source tables so no impact on the source database but minimal impact on I/O [recommended and mostly used approach].
4. This is the most recent method and very few solutions are using it. In this method, the data is read at the application level, in other words when the Application writes in Database, at the same time it pushes the operation for CDC processing. No impact on source database or I/O.

ESP (Event-Stream Processing) is a term used along with CDC and Real-Time processing.

As we touched based on Real-Time processing in another topic, ESP is exactly what Real-Time is all about. In other words, the moment any activity has happened in source system or in source application which can be online portal, social media or a mobile etc., so the moment any activity takes place, the latest action is pushed to next stage. Next stage can be a Data Lake or Data Warehouse, or it can be intermediately solution in in-memory to prepare a response to send directly back to customer’s mobile.

‘Even-Streaming Processing is heavily used if a reaction is planned in Real-Time.’

For example, a customer buys a product from a shop in a mall, the moment he or she use their credit card for payment, along with OTP code another message comes to their mobile, telling them there is a sales or promotion going on in this mall, please have a look before making this payment. This is called Real-Time Even-Streaming Processing.

Data Virtualization is the virtual consumption layer and connect with anonymous data stores i.e., RDBMS, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. As it’s connecting directly with all data stores, the data models of all systems remain intact with no change.

‘It’s considered as one of the solutions for Data Sovereignty problem statement.’

What data stores?

Here data stores are referred as traditional RDBMS or OLTP systems, OLAP systems, Data Warehouse, Data Lake, Delta Lake, Data Vault, Business Vault, Kafka, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. Yes, it’s confusing as we all know above data stores serve their purpose for one or the other business requirement then why this another layer on top?

Data Virtualization hosts views connecting directly with all possible data stores. These views can be one-to-one or can be complex SQL written views to pull data from multiple source systems to create a virtualized layer on top of consumer layer to consume.

Data Federation works just like Data Virtualization framework but what is the difference between both.

Relationship between Data Federation and Data Virtualization is just like relationship between Data Mining and Data Science (there is separate topic on Data Mining vs Data Science).

Conceptually, both Data Federation and Data Virtualization works the same way, but Data Federation is restricted to relational database whereas Data Virtualization also includes data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc., as well.

Another major difference i.e., Data Federation doesn’t create a unified data model whereas Data Virtualization does.

Below are a few similar behaviors of both Data Federation and Data Virtualization.

• Single Access Point: Access all data stores from a single environment.
• UAM: Use Access Management at one place.
• Data Quality Management: Data Quality issues can be identified from a single point.
• No Data Movement: No ETL/ ELT is required.
• Ease of use: As users don’t have to write queries in different languages. Users can write in one language, and it will create a wrapper to talk to all the connected systems.
• No Additional Storage: As a virtual copy there is no cost for any separate copy of data, so no additional storage is required.
• Low Risk: As it’s not hosting the data copy as compared to Data Warehouses or Data Lake where data is physically moved, there is no chance of missing any data.
• Better for Analytics: As all the source data models are connected and data is fetched in real like, it makes source data available in real-time for Data Analysts and Data Scientist
• Performance Impact: Some impact on source systems as some queries will run at source.
• Functional Knowledge: Seek dedicated involvement by Data Stewards and Functional consultants.
• No Historical Data: As there is no copy of data hosted so history data is also not maintained. These are best suited along with Data Warehouse in-place.

Data Virtualization, Data Federation and Data Consolidation all are discussed together with respect to exposing data to data consumers.

Where Data Virtualization and Data Federation do not host any copy of data in its environments, Data Consolidation on the other hand keeps complete historical copies of all the data coming from all the source systems. In this process sometimes, a complex ETL or ELT takes place to process data in different data models and sometimes one to one data model is maintained.

Yes, I am referring to Data Warehouses, Data Marts, Data Lakes, Delta Lakes, Data Vault, Business Vaults etc., all these data stores fall under the umbrella of Data Consolidation environments.

Data Encoding and Decoding is a known term but is always confused with Encryption and Decryption.

Just like Encryption and Decryption, Encoding and Decoding convert and revert data from another format using Key.

Importance of Keys

• Encryption and decryption conversion KEYS are used by known entities or individuals.
• Encoding and decoding KEYS are used by unknown entities or individuals i.e., publicly.

Data Encoding and Decoding is normally used by programs or systems internally whereas Encryption and Decryption are used by proprietary expensive software(s).

There are many encoding techniques but major four types are.

1. Analog data to Analog Signals: This technique is used when sender and receiver both are Analog compatible like mic to speakers directly.
2. Analog data to Digital Signals: This technique is used sender is Analog, and receiver is Digital compatible like mic to computer for recording purpose.
3. Digital data to Analog Signals: This technique is used when the sender is Digital, and receiver is Analog compatible like from computer to speakers.
4. Digital data to Digital Signals: This technique is used when sender and receiver both are Digital compatible like data transfer from computer to computer.

This term is normally known by the Security department and is tagged with Hackers.

‘Data Scraping is also known as Web Scraping.’

The minor difference between both terms is Data scraping means searching the location of the data and then extracting it. In other words, it does not copy and paste rather extract data in a precise and accurate manner whereas Web scraping involves extracting exactly everything from web pages.

Data Scraping or Web Scraping is the process of importing information from a website into a spreadsheet or local file saved on your computer.

Popular uses of data scraping include:

• Research for web content/business intelligence.
• Pricing for travel booker sites/price comparison sites
• Finding sales leads/conducting market research by crawling public data sources (e.g., Yell and Twitter)
• Sending product data from an e-commerce site to another online vendor (e.g., Google Shopping)

It allows us to take unstructured, scattered data from multiple sources and collect it in one place and make it structured for better use.

A Logical Data Warehouse (LDW) is not a new storage technology but an architectural approach to delivering a unified data warehouse view without physically consolidating all data into one centralized repository. Unlike a traditional physical data warehouse, which extracts, transforms, and loads data into a dedicated storage layer, a Logical Data Warehouse integrates data virtually. It connects distributed data sources and presents them as a single, consistent analytical environment.

The concept does not replace traditional modeling approaches. A Logical Data Warehouse can still support third normal form structures aligned with the philosophy of Bill Inmon or dimensional modeling approaches such as star and snowflake schemas aligned with Ralph Kimball. The distinction lies not in the modeling technique but in the physical location of the data. In an LDW, data may remain in source systems, data lakes, physical warehouses, business vault layers, delta-based platforms, streaming systems, or APIs. The integration happens logically rather than physically.