Knowledge Hub

Data Virtualization is the virtual consumption layer and connect with anonymous data stores i.e., RDBMS, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. As it’s connecting directly with all data stores, the data models of all systems remain intact with no change.

‘It’s considered as one of the solutions for Data Sovereignty problem statement.’

What data stores?

Here data stores are referred as traditional RDBMS or OLTP systems, OLAP systems, Data Warehouse, Data Lake, Delta Lake, Data Vault, Business Vault, Kafka, data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc. Yes, it’s confusing as we all know above data stores serve their purpose for one or the other business requirement then why this another layer on top?

Data Virtualization hosts views connecting directly with all possible data stores. These views can be one-to-one or can be complex SQL written views to pull data from multiple source systems to create a virtualized layer on top of consumer layer to consume.

Data Federation works just like Data Virtualization framework but what is the difference between both.

Relationship between Data Federation and Data Virtualization is just like relationship between Data Mining and Data Science (there is separate topic on Data Mining vs Data Science).

Conceptually, both Data Federation and Data Virtualization works the same way, but Data Federation is restricted to relational database whereas Data Virtualization also includes data appliance, NoSQL, Web Services, SaaS, and enterprise applications etc., as well.

Another major difference i.e., Data Federation doesn’t create a unified data model whereas Data Virtualization does.

Below are a few similar behaviors of both Data Federation and Data Virtualization.

• Single Access Point: Access all data stores from a single environment.
• UAM: Use Access Management at one place.
• Data Quality Management: Data Quality issues can be identified from a single point.
• No Data Movement: No ETL/ ELT is required.
• Ease of use: As users don’t have to write queries in different languages. Users can write in one language, and it will create a wrapper to talk to all the connected systems.
• No Additional Storage: As a virtual copy there is no cost for any separate copy of data, so no additional storage is required.
• Low Risk: As it’s not hosting the data copy as compared to Data Warehouses or Data Lake where data is physically moved, there is no chance of missing any data.
• Better for Analytics: As all the source data models are connected and data is fetched in real like, it makes source data available in real-time for Data Analysts and Data Scientist
• Performance Impact: Some impact on source systems as some queries will run at source.
• Functional Knowledge: Seek dedicated involvement by Data Stewards and Functional consultants.
• No Historical Data: As there is no copy of data hosted so history data is also not maintained. These are best suited along with Data Warehouse in-place.

Data Encoding and Decoding is a known term but is always confused with Encryption and Decryption.

Just like Encryption and Decryption, Encoding and Decoding convert and revert data from another format using Key.

Importance of Keys

• Encryption and decryption conversion KEYS are used by known entities or individuals.
• Encoding and decoding KEYS are used by unknown entities or individuals i.e., publicly.

Data Encoding and Decoding is normally used by programs or systems internally whereas Encryption and Decryption are used by proprietary expensive software(s).

There are many encoding techniques but major four types are.

1. Analog data to Analog Signals: This technique is used when sender and receiver both are Analog compatible like mic to speakers directly.
2. Analog data to Digital Signals: This technique is used sender is Analog, and receiver is Digital compatible like mic to computer for recording purpose.
3. Digital data to Analog Signals: This technique is used when the sender is Digital, and receiver is Analog compatible like from computer to speakers.
4. Digital data to Digital Signals: This technique is used when sender and receiver both are Digital compatible like data transfer from computer to computer.

This term is normally known by the Security department and is tagged with Hackers.

‘Data Scraping is also known as Web Scraping.’

The minor difference between both terms is Data scraping means searching the location of the data and then extracting it. In other words, it does not copy and paste rather extract data in a precise and accurate manner whereas Web scraping involves extracting exactly everything from web pages.

Data Scraping or Web Scraping is the process of importing information from a website into a spreadsheet or local file saved on your computer.

Popular uses of data scraping include:

• Research for web content/business intelligence.
• Pricing for travel booker sites/price comparison sites
• Finding sales leads/conducting market research by crawling public data sources (e.g., Yell and Twitter)
• Sending product data from an e-commerce site to another online vendor (e.g., Google Shopping)

It allows us to take unstructured, scattered data from multiple sources and collect it in one place and make it structured for better use.

The Data Hub is a relatively new term in the evolution of enterprise data architecture, appearing after the rise of the Data Lake. It is important to clarify from the outset that a Data Hub is not another type of storage system, nor is it a new analytical framework. Instead, it plays a distinct role as the central integration and distribution point that connects analytical platforms with operational systems.

To understand why Data Hubs matter, it helps to recall the current division of enterprise systems:

• Operational systems (OLTP): These are core enterprise applications such as ERP, CRM, and core banking systems.
• Data Warehouses (DWH): These serve business intelligence needs such as dashboards, reports, and data mining.
• Data Lakes: These enable advanced analytics such as data science, machine learning, and deep learning.

This division creates a practical problem: how can the results of business intelligence and analytics, KPIs, predictions, classifications, and insights, be shared back with the very applications that drive day-to-day business?

The concept of the Data Fabric is often confused with the Data Hub, but while the two share the goal of breaking down silos, they operate at different levels of abstraction. The Data Hub serves as the distribution and operationalization layer, enabling insights from analytical platforms to flow back into enterprise applications. The Data Fabric, by contrast, is a broader architectural paradigm that unifies data governance, integration, and orchestration across the entire enterprise ecosystem.

In the earlier discussion of the Data Hub, we established that enterprise systems are typically divided into three categories:

• Operational systems (OLTP): core business applications such as ERP, CRM, and core banking.
• Data Warehouses (DWH): supporting business intelligence through dashboards, reports, and data mining.
• Data Lakes: enabling advanced analytics, machine learning, and data science.

The Hub emerged to connect these silos and to push the results of BI and analytics back into operational applications.

Data Wrangling or Data Mangling or Data Munging are used interchangeably and do the same function.

Data Wrangling is something that every ETL developer, Data Engineer, Data Analyst etc. do daily. It is something that is done before Data Mining, Data Science, Business Intelligence or Business Analytics.

In simple words, Data Wrangling is to prepare data for processing. Note, prepare data for processing, not process data. Yes, you heard it correctly, sometimes Data Wrangling techniques are used for permanent data fixes and sometimes to prepare data for specific use cases.

Let's take an example, refer to the tables above, fixing and streamlining data in fields i.e., Name, Phone, Birth Date and State will clean the data permanently which will assist in fast job processing, save analysts time for investigation etc. Secondly, you can see row#5 has been removed as, if the analysis is on Date then Year is missing from row#5 so this row will not assist in modelling.

Normally Data Cleansing and Data Enrichment are confused with each other.

‘Data Enrichment is about enhancing the data whereas Data Cleansing is about correcting the data.’

Data Cleansing has been covered in one of the previous topics, so we will focus only on Data Enrichment. As the definition explains itself, Data Enrichment is about adding more data into your first-hand data set from other internal or third-party data sources to add additional context. In this era of social media and unstructured sources, there is always additional data available which can enhance the context of your data generated from internal systems.

‘Data Enrichment must depend on Business Goals.’

For example, if a brand wants to open a new branch, it can buy geographical data from third parties. If a bank wants to offer Credit Cards deals, it can buy crediting rating data from third parties.

Please note, there can be many sources of data sets for Data Enrichment and of course not all can be Enriched in your data stores. Just like any other implementation, Data Enrichment must follow business requirements.

These terms are the core of the Data Pipeline and are normally confused by each other.

Data Integration is to connect with one or more source systems to pull data whereas Data Blending is to merge all datasets into one viewable and understandable presentation view to answer business KPIs. In other words, Data Integration is the prerequisite process for Data Blending, if you can't get data from source systems, you can't blend it to present management for decision making.

Don't forget either Data Science, Business Intelligence or any kind of Analytics all work toward ONE BIG FAT REALITY, to achieve one or the other Business KPI(s).

Coming back to blending and integration. Don't forget in an organization, we have many systems with different objectives where any ONE kind of data could be residing in different patterns in different systems. For example, a product can be sold via multiple channels like from a shop or an online website or maybe from a mobile application etc. When product data is pulled from all these systems in one Data Warehouse solution to report product selling behavior, it's of utmost importance to gather all product selling transactions from all systems into one system which in our example is a Data Warehouse.

Data Mashup is normally confused with Data Mesh (explained separately), but both are totally different.

Data Mashup comes from the idea of doing self-service on a single source of truth. Dashboard or a Web-Page data point coming from multiple sources in a single view is called Data Mashup. Yes, nowadays, we all do this daily.

We have dashboards with data coming from multiple sources which can be Databases, Data Lakes, social media etc. In the Big Data era, we have graphs, pictures, videos all on one page. At the organization level it’s also called Enterprise Mashup. It is also very useful for Data Analysts to fetch data from different sources, make sense of the data and build meaningful data for business decision-making.