Knowledge Hub

While the objective is to imbue data with context, the crucial question is: what type of context are we seeking? Ideally, we aim for a trustworthy and secure context.

Outlined below are 3 high-level steps to guide you in selecting mandatory metadata effectively.

1. Choosing the Right Metadata using the 5W 1H Method: Employ the 5W 1H method (Who, What, When, Where, Why, and How) to meticulously choose metadata that aligns with your specific objectives. This method ensures a comprehensive understanding of the context each metadata element provides.
2. Filtering the metadata with 3 Selection Criteria: Streamline your selection further by applying three essential criteria to the metadata options:

• For Everyone: Choose metadata that anyone can understand, not tied to specific organizations or industries. This makes data easy to use, keeps it relevant, and encourages more people to get involved.
• Anyone Can Create: Pick metadata that anyone familiar with the organization's data can create. No need for technical or legal expertise. This makes the process open to everyone, speeds things up, and promotes collaboration.
• Follow Standards: Ensure your metadata follows universal standards. This keeps things consistent internally and aligns with best practices in the industry, making it easy to share data with others.

What does "success" mean, and why is it so important to have a clear idea of it when dealing with data solutions?

Success is like reaching a goal or doing something well. When we talk about data solutions, which are ways to manage and use information, it's crucial to know what success looks like. Imagine trying to win a game without knowing the rules or score. Defining success in data solutions helps us understand if we are doing things right, meeting our goals, and making the most of our information. It is like having a map to guide us on the path to doing a great job with data.

Let's break it down further:

• Understanding Success: Success, in general, means achieving a goal or doing something well. It's like hitting a target or accomplishing what you set out to do.
• Applying Success to Data Solutions: In the context of data solutions, we're talking about ways to handle and use information effectively. This could include tasks like collecting, organizing, and analyzing data.
• Importance of Defining Success: Think of it as setting clear rules for a game or knowing the score you want to achieve. Defining success in the world of data solutions helps us clarify what we're trying to achieve and how we'll know when we've achieved it.
• Setting Goals: When you define success for data solutions, you're essentially setting goals. These goals could be related to accuracy, efficiency, security, or other aspects of managing data.
• Monitoring Progress: Knowing what success looks like allows you to keep track of how well you're doing. It's like having a map that shows you where you are on your journey and how far you are from your destination.
• Making Informed Decisions: With a clear definition of success, you can make informed decisions. You know what you're aiming for, so you can choose the best strategies and tools to get there.
• Continuous Improvement: Defining success isn't a one-time thing. It's an ongoing process. You can adjust your goals as needed and keep improving your data solutions over time.

Yes, you are right, you might not have heard about this approach. It is something new which Myself is working on.

Let’s decode it…..

Productionizable Viable Product (PVP): A Revolutionary Approach to reduce TTM (Time to Market) of adopting cutting-edge Data & Analytics Technologies while minimizing CAPEX (Capital Expenditure).

‘PVP focuses solely on new data & analytics initiatives rather than enhancements.’

Productionizable Viable Product (PVP) approach represents a distinctive approach in optimizing IT project execution by employing a singular Production environment throughout the project lifecycle. This streamlined architecture is designed to alleviate the complexities associated with managing multiple environments. To address data privacy concerns, the PVP architecture integrates controlled measures such as data masking or anonymization during the initial phases of development and testing, ensuring adherence to regulatory standards. The incremental provisioning strategy is a pivotal aspect, commencing with the provision of minimum required services or tools in the Production environment. The architecture emphasizes iterative testing cycles within the Production environment, allowing for continuous validation of integrated code sets through incremental Testing. This cyclic Testing in Production ensures thorough scrutiny before the final release to end users. In essence, the PVP architecture provides a structured and controlled framework, optimizing workflows, reducing complexities, and enhancing the reliability of Analytics project execution.

CDC framework is much more than CDC as a feature or as a capability by a tool. There are many tools in the market which can just extract CDC data from database which is only the first step in whole CDC framework.

CDC: Full abbreviation of CDC is Change Data Capture, meaning the moment, the data is changed in database ONLY THE CHANGED DATA gets picked up by the CDC process and pushed to next layer. Changed data can be like 1) a new row can be added or 2) an existing row can be updated or 3) a previously added row can be deleted or dropped.

This whole process can be Real-Time but mostly can't as Real-Time + CDC needs a lot of memory processing which comes with heavy TCO, so it all depends on ROI for the related Business Case.

CDC Framework: Extracting only changed data is not the end of settling data to be presented in Dashboards or Reporting, there are two more steps one must complete before changed data is considered useful.

Approaches play a vital role in the failure of many projects in the IT World. We all have heard about POC, Protype, Pilot, MVP, SIT, UAT, and Production implementations, but I am sure, most of us might have not given a thorough thought of aftereffects, if one is interchanged with other.

Let’s try to understand everything in the upcoming topics but to give a broader perspective all above approaches have their own implementation strategies. Most of the projects fail when e.g., started with a POC and once the POC is successful, move the same in Production. One must not do that. This normally happens when business push implementers to achieve tight timelines.

Please note, small level of implementation which in our example, POC might be successful with limited infrastructure or software, but 99% chances are when it goes in Production, it’s bound to fail. Same is true for Prototype, Pilot and MVP environments. One must plan infrastructure and software based on, if plans to be in Production.

In MVP stage where actual Tools and Technologies, and SIT, UAT, and PRODUCTION environments are implemented on a small scale.

Once MVP is successful, the next stage is to gather complete business requirements, a complete designed is made, teams are built, project plan is made, end-to-end SOW is signed, development is conducted, testing is done i.e., UT, SIT and UAT, and solution goes live.

On contrast with POC, Prototype and Pilot/ MVP, in this stage a complete Infrastructure, Software and Data is prepared and used.

UT stands for Unit testing and is done by developer during development phase for every piece of code developed. The better the UT is, the better the product will be.

SIT stands for Systems Integration Testing. Now a days, SIT is disappearing, SIT is the first step to save cost which is NOT a good idea. I am sure in a decade, newbies in IT World might not even know what actually SIT is. Once complete solution is ready, it’s moved to another environment which do have all the possible components of the actual solution but in term of size, it is small e.g., if based on data there is 10 servers and 100 licenses are required, SIT can be run with 1 server and with 1 license. In SIT, we make sure code can run end-to-end, user access is properly set, security protocols are in-place, jobs dependencies are in-place etc.

In typical SQL world (Structured Query Language), usually managing and manipulating data efficiently is very crucial for its performance, database lifecycle or DWH scalability. Temporary tables and materialized views are two powerful constructs (objects) that help in achieving both objectives simultaneously. Even though each serves a unique purpose within data operations and data streams. Let´s look at both little closer.

Temporary Tables

We can define them as typical tables which are created within a database session and are only available during the lifetime of that particular session. Lifetime during session is critical as they are designed to store intermediate results, results of window functions for complex calculations or data processing tasks and massive data manipulations.

Temporary tables are essentially useful in scenarios where data needs to be temporarily held and manipulated without affecting the main database tables. Here comes why it is called “temporary”. After your session ends, the temporary table is automatically discarded - same as you run command DROP TABLE, after query results are taken into another step. This way processing ensures no permanent database storage is consumed. You can imagine storing all 245 steps and their output tables consisting of 6 billion rows and 700 columns each day to have versions of each step.

At its core, functional testing involves validating all the use cases and requirements outlined in the Functional Requirements Document (FRD) or other business requirement specifications to ensure the product operates as intended. In contrast, non-functional testing focuses on evaluating the product's quality attributes, such as performance, compliance with governance standards, security measures, and overall system reliability, ensuring it meets predefined benchmarks beyond functional correctness.

Testing is a critical phase in the software development lifecycle, ensuring that a product meets its requirements and functions as intended. It is broadly divided into two categories: functional testing and non-functional testing. Understanding these categories is essential for developing robust, reliable, and user-friendly software systems. This chapter explores these two testing types, their significance, methodologies, examples, and key differences.