Knowledge Hub

New name of Pilot is MVP 😊. MVP is the next stage of Prototyping where Tools and Technologies are tested, and end solution is tested by technical folks. Now in MVP, subset of actual requirements is considered, subset of real data is used, subset of hardware and infrastructure is used, subset of software is used, and limited number of features are implemented.

Once MVP is ready, it’s released to actual users to start using it. This gives business users a taste of the actual solution. The biggest advantage of this approach is, with small implementation, with small investment, business can taste the blood and take a decision to go to the next stage or drop the solution.

Now a days, many organizations divide complete solution into multiple MVP(s) i.e., by implementing solutions in chunks, this is also supported by Agile approach where product gets ready in shorter timeframes and then keeps maturing with time. In MVP, just like actual implementation, three environments are spin i.e., DEV, UAT and PROD, along with all the steps like DevOps, CI/CD, Metadata Management, Data Quality, Data Governance, Data Lineage, Master and Reference Data Management, and many other steps which are required to go to Production.

In MVP stage where actual Tools and Technologies, and SIT, UAT, and PRODUCTION environments are implemented on a small scale.

Once MVP is successful, the next stage is to gather complete business requirements, a complete designed is made, teams are built, project plan is made, end-to-end SOW is signed, development is conducted, testing is done i.e., UT, SIT and UAT, and solution goes live.

On contrast with POC, Prototype and Pilot/ MVP, in this stage a complete Infrastructure, Software and Data is prepared and used.

UT stands for Unit testing and is done by developer during development phase for every piece of code developed. The better the UT is, the better the product will be.

SIT stands for Systems Integration Testing. Now a days, SIT is disappearing, SIT is the first step to save cost which is NOT a good idea. I am sure in a decade, newbies in IT World might not even know what actually SIT is. Once complete solution is ready, it’s moved to another environment which do have all the possible components of the actual solution but in term of size, it is small e.g., if based on data there is 10 servers and 100 licenses are required, SIT can be run with 1 server and with 1 license. In SIT, we make sure code can run end-to-end, user access is properly set, security protocols are in-place, jobs dependencies are in-place etc.

CRUD and CRAP functions are our daily used processes for data folks, but surely by name most are not aware of these words 😊.

Let’s decode these…..

As we have been using structured data for the last few decades, we know how to create, read, update, and delete. These basic functions which we use in any database are called CRUD data.

Now, the new era, the current era, surfaced where data started to flow-in with high velocity from web logs, social streams, CCTV, sensor data, videos, Facebook, or Tweeter data, mobile, geo-spatial, and so on. All this data has nothing to do with transactional data, so we used to call it CRAP data 😊. But now these have become one of the most valuable assets of any organization. The speed this data is coming, it’s normally not possible to update or delete it so it’s been created, replicate, appended, and process only.

CRUD stands for Create, Read, Update, and Delete

Accessibility and Visibility are often used interchangeably, especially in simple terms. However, in the world of data solutions, it's crucial to distinguish between them.

Let’s decide it…..

Accessibility pertains to your ability to connect to and utilize data for decision-making purposes.

On the other hand, Visibility involves having a comprehensive understanding of what's happening behind the scenes with your data.

For instance, if you request your Data Warehousing or Data Lakehouse datastore owners to extract data from source systems and transform it into the desired format for decision-making, two key concepts come into play:

Accessibility: This is achieved when your database administrator or datastore owner grants you privileges within the database, allowing you to read, write, or update your data.

Visibility: This extends beyond mere access. It encompasses a deep understanding of various aspects of your data, such as what was sourced, the business rules used for transformations, the presence of business, technical, or operational metadata, the formula for golden records, data quality assurance methods, data loading approaches (e.g., CDC, batch, real-time), SLAs, and lineage information, among other things. All of these elements collectively provide you with comprehensive Visibility into your data.

Generative Artificial Intelligence (AI) has become a prominent trend in recent times, particularly emphasized by our enthusiastic Sales folks 😊. But let's pause for a moment, are you familiar with the concept of Synthetic Data? Surely, it's not new to you 😊. And for those with a tech-savvy mindset, the term "Generative AI" likely triggers thoughts of Generative Models or Statistical Models, perhaps with a knowing smile.

Indeed, the application of generative and statistical models has been around for decades. In fact, the idea of Generative AI isn’t new at all, it dates back to as early as 1950, when Alan Turing proposed machines that could exhibit intelligent behavior.

Today, however, we're witnessing a resurgence of interest, largely due to the accessibility and power of tools like OpenAI’s ChatGPT, Google Gemini, Anthropic’s Claude, Meta's LLaMA, and Stability AI's Stable Diffusion. These tools represent a leap forward in putting generative capabilities into the hands of everyday users, generating text, images, audio, and even code with natural ease.

ETLT is a way of talking about how we handle information. Think of it like this: first, we take out the data we need. Then, we change it to make sure it's clean and ready to use. After that, we put it where we want it to be. Lastly, we might make some more changes to the data to make it even better for understanding. ETLT is like a helpful addition to the usual process (ETL) that makes our data even more useful and organized.

ETLT stands for "Extract, Transform, Load, and Transform" in the context of data integration and processing. Please don’t worry about where this ETLT has come from. ETLT just includes another stage in ETL world to support semantic layers.

Let’s understand each:

• Extract: Involves extracting data from various sources, which could be databases, applications, logs, or other systems.
• Transform (First T): Refers to the process of cleaning, structuring, and enriching the extracted data so that it is suitable for analysis. Transformation may include data cleansing, normalization, aggregation, and other operations.
• Load: Involves loading the transformed data into a target database or data warehouse where it can be used for reporting, analysis, or other purposes.
• Transform (Second T): This additional transformation step might be included in some interpretations of ETLT. It represents further processing or transformation that occurs after the data has been loaded into the target system. This step acknowledges that data transformation can be an ongoing and iterative process.

In the world of data, synchronization and propagation are like superheroes that make sure our information stays accurate and consistent, even when it's spread out in different places.

Let's understand these ideas:

Synchronization:

• Synchronization is about keeping data the same and up to date in many different spots or systems.
• The goal is to ensure that if we change something in one place, like updating the inventory in an online store, those changes quickly show up everywhere else, such as on the website and in the physical store's system. This way, everyone using the data gets the latest and correct info.
• For example, imagine you run an online store, and you add new items to your inventory on your computer. Synchronization is like magic that makes sure those new items appear on your website and in your store's computer, so customers and the store both know what's available.

Propagation:

• Propagation is like the spreading of news, but for changes in data. It's about making sure updates in one part of a system travel to other connected parts.
• When something changes, like depositing money at an ATM in a bank, propagation helps that change reach all the places that need to know, such as your online bank account and the main bank database. It's like making sure the news of the deposit goes to all the right places.
• For example, if you put money into an ATM, propagation ensures that the information about your deposit reaches all the important places, like your online bank app and the bank's main computer. It's like making sure everyone in the banking system knows about the deposit.

In data modelling, particularly in the context of relational databases, the terms "supertype" and "subtype" are used to describe the relationships between entities.

Let's explore these concepts:

Supertype: A supertype is a generalization that represents a higher-level entity. It encompasses common characteristics shared by multiple lower-level entities.

• It contains attributes common to multiple related entities.
• It represents the generalization of several entities.
• For example, consider a supertype "Vehicle" that includes common attributes like "Manufacturer," "Model," and "Year." Cars, motorcycles, and trucks could be subtypes of this supertype, inheriting these common attributes.
• Subtype: A subtype is a specialization of a supertype, representing a more specific entity with additional attributes or characteristics.
• It inherits attributes from the supertype and may have additional attributes.
• It represents a more specific category within the generalization.
• For example, using the "Vehicle" example, "Car" is a subtype that inherits attributes from the "Vehicle" supertype but may have additional attributes specific to cars, such as "Number of Doors" or "Body Type."

ISO 55000, 55001, 550012 and 550013 are standards that provide guidance and best practices for managing assets effectively within an organization.

Let's break down these standards in simpler terms:

• ISO 55000 – A comprehensive series of international standards, plays a pivotal role in guiding organizations toward effective asset management practices. Within this framework, the concept of treating data as a valuable asset is gaining increasing prominence. ISO 55000 underscores the significance of data in asset management by emphasizing its role in decision-making processes and the overall lifecycle of assets. Organizations are encouraged to recognize data as a strategic asset, acknowledging its potential to enhance decision quality, mitigate risks, and optimize asset performance.
• ISO 55001 – a core component of the ISO 55000 series, provides a comprehensive framework for organizations to establish, implement, and continually improve their asset management systems. Within the context of ISO 55001, the recognition of data as a crucial asset is paramount. This standard encourages organizations to integrate data management practices into their asset management processes, acknowledging the pivotal role that data plays in decision-making and overall asset performance. ISO 55001 emphasizes the need for systematic approaches to managing data throughout the entire asset lifecycle, from acquisition to disposal. It underscores the importance of ensuring data accuracy, reliability, and relevance, as these factors significantly impact the effectiveness of asset management decisions. The standard promotes the establishment of robust data governance and information management processes, ensuring that data is treated with the same level of care and attention as other physical or financial assets. By incorporating data as a strategic asset within the ISO 55001 framework, organizations can harness the power of information to optimize asset performance, minimize risks, and enhance the overall value derived from their assets. This integration of data management principles into the asset management system aligns with contemporary best practices, acknowledging the critical role of data in achieving organizational objectives and sustaining long-term asset value.
• ISO 55012 (New) – The section on "Guidance on People Involvement and Competence" within the ISO 55000/1 standards serves as a comprehensive exploration of the human dimension in the realm of asset management. Recognizing the expanded scope of ISO 55000/1, which encompasses a broader understanding of asset management, this section specifically delves into the integration of data, information, and knowledge management within the human-centric aspects of the discipline. It provides detailed insights into how individuals within an organization, spanning various roles and levels, contribute to the effective implementation of asset management practices. This includes not only their technical competence but also their understanding and management of data, information, and knowledge as integral components supporting the overarching goals of asset management. By acknowledging the human-side dynamics, this guidance emphasizes the need for a skilled and knowledgeable workforce capable of leveraging data and information to make informed decisions, optimize asset performance, and ultimately contribute to the success of the asset management framework outlined in ISO 55000/1. In essence, it underscores the interconnectedness of human capabilities, data management, and achieving excellence in asset management practices.
• ISO 55013 (New)– The section titled "Guidance on the Management of Data Assets" represents a comprehensive and detailed examination of the pivotal role that data plays within the context of asset management. This segment is structured into primary clauses, each focusing on a distinct aspect of managing data as a valuable asset. The clauses include "Managing Asset Data," which outlines strategies for effectively handling data throughout its lifecycle; "Delivering Value from Asset Data," which underscores the importance of extracting meaningful insights and value from the data; "Identifying Data Assets," which guides organizations in recognizing and categorizing their data assets; "Managing Data Assets," providing insights into best practices for the ongoing management and maintenance of data; and "Governance," which emphasizes the need for a robust governance framework to ensure the integrity, security, and compliance of data assets.

ISO 11179 is an international standard that provides guidelines for the representation and management of metadata for describing data and information resources. In simpler terms, it offers a standardized way to organize and describe data, making it easier to understand, share, and manage.

Let's break down ISO 11179 in more detail:

As we have read in Metadata Management topic, Metadata is Data about Data. It describes the characteristics, properties, and context of data, helping users understand what the data is, how it was collected, and how it can be used.

ISO 11179 focuses on standardizing the way metadata is defined, organized, and managed. It provides a common framework for describing data elements, ensuring consistency and clarity in data descriptions. The standard helps organizations create a common language and structure for describing data, making it easier for different systems and people to understand and use the data.