Knowledge Hub

Data Discrepancies are confused with Data Errors which is not true. Data Errors are searched, identified, and fixed based on Data Discrepancies.

‘Dictionary: An illogical or surprising lack of compatibility or similarity between two or more facts are called discrepancies.’

Data discrepancies can be like mismatch of data from source verses data processed.

Some time, when data analysts run manual calculation, those are not matching with the numbers coming from automated reports which means there are data reconciliation issues. This is Data Discrepancy.

For example, two results do not match based on business logic. Refer to the image, total of 9,093 + 6,104 is not equal to All Users i.e., 14,018. This means, a total of 14,018 came from various sources and 9,093 & 6,104 came from different sources. This is Data Discrepancy.

As mentioned above, once data discrepancies are found, the next step is to start identifying the data errors, code errors, business rule errors etc.

Coupling is a parameter dependency approach used cross software or cross systems.

This approach is used when there are more than one software or system is in the picture to deliver the result.

Data Coupling is about when data elements are used, to kick-off the next step in a different system. For example, in data pipeline, we are pulling data from a hospital management system e.g., when a baby is born let’s have a data dependency like, the moment the baby is registered in system, pull its data. So, it’s a cross-system data dependency or we called Data Coupling.

The disadvantage of Data Coupling approach is, if something goes wrong in the first system, dependent systems cannot move.

Let me share another old example. Old data pipeline folks will remember this one i.e., when we used to load files from another system, we had to wait till the time whole file movement is completed. The challenge was how to make sure when to start pulling like only once file is completely copied.

Data Imputation is majorly used in Data Science domain.

‘Imputation means, fill missing data.’

Data Imputation normally comes into play so business rules can be imposed on all data rather on subsets of data. Before Machine Learning Model runs, data imputation is done by either missing data is filled up or the whole row is removed.

Missing value can create many kinds of issues e.g., during categorizing, classifying, clustering etc., if data is missing then results will not be 100%.

Generally, in numeric datasets, missing values are replaced with MEAN values. Having said that, there many ways to deal with missing data.

Few Imputations Techniques

• The first technique is where data is missing at random basis but before using it, it is completely removed from the dataset.
• The second technique is also where data is missing at random basis but here missing values are replaced referring to the other data points available with current dataset.
• Third technique is where data is not missing at random basis and mainly replaced using Mean/Median/Mode values.

‘Disambiguation means to make something clear and understandable.’

We all know the word i.e., Ambiguous which means something which is unclear especially when same words are used for multiple purposes.

In the current era, semi and unstructured data is the norm. Now a days, social media has become one of the major sources for customer profiling. Then audio and videos are another source which can have any kind of data behavior.

‘Data Disambiguation’s biggest use case is to handle Sarcasms.’

As we know, in Sarcasm, dual meaning words are heavily used. For example, there is a common saying i.e., What’s up? This phrase can be used to say greeting to a friend and can also be used to show anger to someone unknown moving around in your restricted area.

The job for Data Disambiguation process is.

• To dig out the right meaning out of confusing statements.
• To understand the right concept of statement.
• Remove anything which can corrupt the behavior of pattern.
• Translate or replace dual meaning word with single meaning words so end user is not confused before taking any decision.

Data Fusion, another term overlapping with many other already in-place terms.

‘Data Fusion is about pulling same nature of data from multiple source systems to co-relate with each other to produce one unified output.’

For example, data for a particular product can come from POS (point of sales), from shops, from online stores etc. At the same time, people can talk about that product on social media, in seminars, webinars etc. Fusion of all those data elements can produce valuable information about that product for sales and marketing teams.

Yes, looking into image you can make out same thing can be achieved in Data Warehouse, Data Mart, Data Lake, Data Lakehouse, Data Vault/ Business Vault etc., and you are not wrong. All above-mentioned Data Store can be used for Data Fusion.

Integration Hub is normally confused with datastores like Data Warehouse, Data Lake, Data Lakehouse etc., which is not correction.

Integration Hub is a single central place from where you can send instructions to internal and external systems to perform some actions. Integration Hub consists of Spokes, each Spoke consists of number of steps for actions.

Generics Steps

• Create actions to be sent out to external systems.
• Run commands on external servers.
• Allow sending messages e.g., Rest, SOAP etc.
• Allow parsers.
• Allow SQL command to databases.

Yes, we do it every day e.g., we create connections with any system e.g., creating a connection with SharePoint then running commands to Delete a file. Consider 100 users want to do the same action, each must go through the same number of steps. Using Integration Hub, Spokes consisting of connections, credentials, and actions to be performed on e.g., SharePoint. Now, all those 100 users can reuse the same Spoke created to delete files in SharePoint.

Hype Cycle of Data Management is confused with Data Management which is not correct as one is about solution and other about enterprise.

Hype Cycle of Data Management is about knowing the pattern of technologies like how past tools and technologies performed, and which are new technologies in trend.

Hype Cycle of Data Management assist organizations to plan its technologies roadmap, it assists to take decisions like which tools and technologies to adopt and which not to adopt.

There are many a times when organizations are reluctant to adopt new technologies and they end up like Kodak and Nokia.

There are many companies which prepare these kinds of statistics and share with CTO(s), CDO(s), CFO(s), CEO(s) etc.

What does "success" mean, and why is it so important to have a clear idea of it when dealing with data solutions?

Success is like reaching a goal or doing something well. When we talk about data solutions, which are ways to manage and use information, it's crucial to know what success looks like. Imagine trying to win a game without knowing the rules or score. Defining success in data solutions helps us understand if we are doing things right, meeting our goals, and making the most of our information. It is like having a map to guide us on the path to doing a great job with data.

Let's break it down further:

• Understanding Success: Success, in general, means achieving a goal or doing something well. It's like hitting a target or accomplishing what you set out to do.
• Applying Success to Data Solutions: In the context of data solutions, we're talking about ways to handle and use information effectively. This could include tasks like collecting, organizing, and analyzing data.
• Importance of Defining Success: Think of it as setting clear rules for a game or knowing the score you want to achieve. Defining success in the world of data solutions helps us clarify what we're trying to achieve and how we'll know when we've achieved it.
• Setting Goals: When you define success for data solutions, you're essentially setting goals. These goals could be related to accuracy, efficiency, security, or other aspects of managing data.
• Monitoring Progress: Knowing what success looks like allows you to keep track of how well you're doing. It's like having a map that shows you where you are on your journey and how far you are from your destination.
• Making Informed Decisions: With a clear definition of success, you can make informed decisions. You know what you're aiming for, so you can choose the best strategies and tools to get there.
• Continuous Improvement: Defining success isn't a one-time thing. It's an ongoing process. You can adjust your goals as needed and keep improving your data solutions over time.

Yes, you are right, you might not have heard about this approach. It is something new which Myself is working on.

Let’s decode it…..

Productionizable Viable Product (PVP): A Revolutionary Approach to reduce TTM (Time to Market) of adopting cutting-edge Data & Analytics Technologies while minimizing CAPEX (Capital Expenditure).

‘PVP focuses solely on new data & analytics initiatives rather than enhancements.’

Productionizable Viable Product (PVP) approach represents a distinctive approach in optimizing IT project execution by employing a singular Production environment throughout the project lifecycle. This streamlined architecture is designed to alleviate the complexities associated with managing multiple environments. To address data privacy concerns, the PVP architecture integrates controlled measures such as data masking or anonymization during the initial phases of development and testing, ensuring adherence to regulatory standards. The incremental provisioning strategy is a pivotal aspect, commencing with the provision of minimum required services or tools in the Production environment. The architecture emphasizes iterative testing cycles within the Production environment, allowing for continuous validation of integrated code sets through incremental Testing. This cyclic Testing in Production ensures thorough scrutiny before the final release to end users. In essence, the PVP architecture provides a structured and controlled framework, optimizing workflows, reducing complexities, and enhancing the reliability of Analytics project execution.

A Data Abstraction Layer (DAL) is like a special helper in computer science and software engineering. It's a layer of code that keeps the tricky details of how data is stored or used, separate from the rest of the software.

Here are some important things to know about a Data Abstraction Layer:

• Abstraction: It means hiding all the complicated stuff about storing data behind a simpler way of working with it. This makes it easier for developers to handle data without worrying too much about the nitty-gritty details.
• Separation of Concerns: The Data Abstraction Layer helps keep things organized by separating how data is stored from how the software works. This makes the software easier to understand and take care of.
• Consistent Interface: The layer provides a set of clear rules that the rest of the software follows to work with the data. This is helpful because it keeps the software running smoothly even if the way data is stored changes.
• Adaptability: The Data Abstraction Layer acts like a shield, protecting the software from changes in how data is stored. So, if the way data is kept in the computer changes, only the Data Abstraction Layer needs to be adjusted, and the rest of the software can stay the same.
• Security and Validation: The Data Abstraction Layer also looks out for the security of the data. It makes sure that data is handled in a safe and controlled way.