Knowledge Hub

As much as Data Filtering is simple as much as it’s so important in Data World.

‘Data Filtering is about reducing the number of rows while pull data from a table.’

On Cloud: In Cloud, resources i.e., compute power or RAM are just click away. In fact, now a days these can automatically increase based on your query requirements. So, if you are using Cloud then, STRICKLY, no one is allowed to run any query either in PROD, UAT or Dev environment without any filter, means in a SQL command one must use WHERE clause as each query is attached with a dollar value. In Cloud, filtering becomes THE FACTOR to maintain resources utilization at minimum, considering as mentioned each resource costs a dollar value.

On-Premises: In contrast of Cloud, resources i.e., compute power or RAM in on-premises systems is limited. To increase resources on premises there is a need for a complete buying cycle by procurement which takes months. With static resources, if someone runs a query without a filter, that query can hold onto all the available resources, locking the rest of the queries. This can create delays in providing other datasets to top management, means delays in decision making, means losing business opportunities, means loss and no one like losses right. In nutshell, in on-premises solutions, filter also is very critical. At the same time developers has the liberty to us any kind of query in development environment as size of data is limited and static.

Syndication is also known as synchronization, yes now it must be making sense to you 😊.

‘Data Syndication is a process not a technology.’

The word syndicate comes from the French word syndicate which means "trade union" (syndic meaning "administrator"), from the Latin word syndicus which in turn comes from the Greek word σύνδικος (syndikos), which means "caretaker of an issue"; compare to ombudsman or representative.

‘Data Syndication is to collect and prepare update-to-date dataset for a particular activity.’

For example, prepare a dataset from multiple source systems for the marketing team to plan their upcoming campaigning exercise.

Data Syndication process

• Can handle any type of format as input and output as well.
• It can send alerts.
• It can configure access matrix.
• It provides monitoring dashboards and reports for compliance and audit trials.
• As it’s a process, any expensive or cheap technology can be utilized.
• Output formats can be designed based on customer requirements.
• Can send email, SMS, spreadsheet, database, ML models etc.

Data Supply Chain is not a data capability rather a process.

Just like generic supply chain, Data Supply Chain works in the same manner where it starts from Source Data Stage 🡪 Transformation and Integration Stage 🡪 Consumption Stage

• Source Data Stage is the first stage where raw data is planned and identified i.e., OLTP systems.
• Transformation and Integration Stage is where data is converted based on business requirement in-shape of Data Lake Systems or Data Warehouses
• Consumption Stage is where data is store for business use in-shape of Data Lake Systems or Data Warehouses (OLAP systems) or Semantic Layer etc.

Data Supply Chain is a very critical operational process to manage Data Asset and Liabilities. In the absence of visibility on Data Supply Chain, organization cannot plan ROI as well as TTM which is critical for the Data Monetization.

Note: Data Assets, Data Liabilities and Data Monetization all are explained in separate topics.

As the name says all, Data Sharing is about making data accessible to all the relevant stakeholders.

‘Data Sharing is about eliminating Dark Data.’

Data Sharing is one of the key components in Data Driven Organization where organizations aim to build data culture with Data Liberalization.

Data Sharing Channels

• Using Business Intelligence tools
• Using languages like SQL, Python etc.
• Using APIs or direction Using JDBC/ ODBC Connectors
• Using built-in Drivers
• Using Machine Learning or Using Data Mining Models
• Exporting in Excel, CSV, FF formats
• Pushing data to other databases and applications
• Letting other databases and application to Pull data.
• ETL/ ELT tools
• Audit, Compliance or Regulatory Reports etc.

Data Fusion, another term overlapping with many other already in-place terms.

‘Data Fusion is about pulling same nature of data from multiple source systems to co-relate with each other to produce one unified output.’

For example, data for a particular product can come from POS (point of sales), from shops, from online stores etc. At the same time, people can talk about that product on social media, in seminars, webinars etc. Fusion of all those data elements can produce valuable information about that product for sales and marketing teams.

Yes, looking into image you can make out same thing can be achieved in Data Warehouse, Data Mart, Data Lake, Data Lakehouse, Data Vault/ Business Vault etc., and you are not wrong. All above-mentioned Data Store can be used for Data Fusion.

Integration Hub is normally confused with datastores like Data Warehouse, Data Lake, Data Lakehouse etc., which is not correction.

Integration Hub is a single central place from where you can send instructions to internal and external systems to perform some actions. Integration Hub consists of Spokes, each Spoke consists of number of steps for actions.

Generics Steps

• Create actions to be sent out to external systems.
• Run commands on external servers.
• Allow sending messages e.g., Rest, SOAP etc.
• Allow parsers.
• Allow SQL command to databases.

Yes, we do it every day e.g., we create connections with any system e.g., creating a connection with SharePoint then running commands to Delete a file. Consider 100 users want to do the same action, each must go through the same number of steps. Using Integration Hub, Spokes consisting of connections, credentials, and actions to be performed on e.g., SharePoint. Now, all those 100 users can reuse the same Spoke created to delete files in SharePoint.

To make it easy, first must understand Traditional Data Stake (TDS). Now, what is TDS lolz 😊. TDS is your On-premises solution which can include any kind of Databases or maybe Hadoop environment as well. Whereas MDS is your tools and technologies on Cloud.

‘MDS vs TDS is more over a comparison between Cloud vs On-premises infrastructures.’

In TDS/ On-premises, if you need new hardware, there is a complete procurement cycle involved. If you need 2 CPU of processing at nighttime and 4 CPU in daytime but you end up paying 4 CPU for 24 hours as you have already bought the hardware 😊.

In MDS, when you are on Cloud, any resource is few clicks away. If you need 2 CPU as night, you only pay when you are using those and same for daytime CPU requirement.

Data Integration is the action of combining data to extract meaningful information and business value.

If we become more technical, data integration is about exchanging content among systems and analyzing it to design decision support systems for business requirements.

Terminology

• A Data Integration Layer is a layer which integrates multiple source systems with target systems.
• A Back end is something which shares data and sources called as source system).
• A Connector is the component which connects source and target so they can share data among each other.
• A Front-End is a target system which utilizes the data for making decisions. It can be OLAP or a digital application.
• An Orchestration is a process in which defines when data will travel from source, when it will be transformed in the shape that target can understand and when it will push to target system.

In the previous diagram:

• The Web Application, the Mobile Application, the Front-End A are Front-Ends.
• The ERP, Legacy Tool, and Back-End B serve as back-end systems.
• To connect to ERP, Legacy tool and Back-End B, those can be connected via C1, C2 and C3, respectively.

Many people mistakenly assume that Integration and Interoperability are interchangeable terms, but they are actually quite different. While they work together, their nature and purpose are distinct.

Let’s decode these…..

Integration involves connecting systems and exchanging data to fulfill a range of objectives [Data Integration explained in different topics]. However, when data is shared between systems, it often requires translation to be understood by the receiving party. Despite the successful integration of systems, these translations conflict with the essence of interoperability.

Interoperability, on the other hand, is the ability to understand data without the need for translation. When two or more systems are linked and exchange data seamlessly without requiring translation for comprehension by the receiving parties, it signifies the presence of interoperability. A notable instance of interoperability is evident in the medical domain, where various hospitals can readily interpret key terms and codes within shared data, whereas an average individual may struggle to do so.

A Data Abstraction Layer (DAL) is like a special helper in computer science and software engineering. It's a layer of code that keeps the tricky details of how data is stored or used, separate from the rest of the software.

Here are some important things to know about a Data Abstraction Layer:

• Abstraction: It means hiding all the complicated stuff about storing data behind a simpler way of working with it. This makes it easier for developers to handle data without worrying too much about the nitty-gritty details.
• Separation of Concerns: The Data Abstraction Layer helps keep things organized by separating how data is stored from how the software works. This makes the software easier to understand and take care of.
• Consistent Interface: The layer provides a set of clear rules that the rest of the software follows to work with the data. This is helpful because it keeps the software running smoothly even if the way data is stored changes.
• Adaptability: The Data Abstraction Layer acts like a shield, protecting the software from changes in how data is stored. So, if the way data is kept in the computer changes, only the Data Abstraction Layer needs to be adjusted, and the rest of the software can stay the same.
• Security and Validation: The Data Abstraction Layer also looks out for the security of the data. It makes sure that data is handled in a safe and controlled way.