Knowledge Hub

DLOps is like MLOps and belongs to Data Science domain.

Before we try to understand DLOps as compared to MLOps, one must understand the difference between Machine Learning and Deep Learning. (There is separate topic on difference between Machine Learning and Deep Learning)

‘It won’t be wrong to say, MACHINE LEARNING is for simpler and more structured & semi-structured use cases whereas DEEP LEARNING is for complex and unstructured use cases.’

Let’s try to understand with a simple example. If we need to find out how many cars, bikes or buses there are in structured or semi-structured datasets then it’s relatively easy as data is in rows and columns format. We can have the width, length, and height of each car, bike or bus so using MACHINE LEARNING models we can easily ready this dataset and produce expected output. But what if we give pictures to ML models to extract the same result, will it be possible? The answer is No as ML models normally work on structured or semi-structured datasets. To find out cars, bikes, or buses from pictures, DEEP LEARNING models are used which can recognize the objects in the pictures and find out which presents car or bike or a bus.

We touched base with Data Mining and Data Science, it will be ideal to discuss Data Dredging, Snooping, p-hacking, and Fishing. New terms right 😊?

Data Dredging, Snooping and Fishing all refer to the same behavior of data analysis BUT without proper hypothesis and relationship among datasets.

Data mining finds results based on the correlation of data in large data sets, but Data Dredging, Snooping, p-hacking, and Fishing find results based on chance methodology. In other words, Data Dredging, Snooping, p-hacking, and Fishing share the results which require more investigation. These sometimes bypass Data Mining techniques and come up with immature conclusions. On occasions, it shows more details about something than it contains.

There are two types of Fact Findings in any analysis that ultimately assists in DSS i.e., 1) known facts 2) unknown facts. Data mining techniques extract Known Facts as it’s done only on structured datasets and mostly, we know what's there in structured data like relationships between tables and relationships of data within the same table. But that's Not True, sometimes, even in structured data there are Unknown Facts and that’s where Data Dredging, Snooping and Fishing techniques come into the action.

There are four types of instructions to pull data i.e., SISD, SIMD, MISD and MIMD.

• SISD: Single Instruction Stream and Single Data Stream
• SIMD: Single Instruction Stream and Multiple Data Stream
• MISD: Multiple Instruction Stream and Single Data Stream
• MIMD: Multiple Instruction Stream and Multiple Data Stream

Before we jump into understanding all above mentioned techniques, let’s first understand what is a Core in a machine or in a computer or in a server etc.?

A CPU Core is built of billions of microscopic transistors to process instructions passed to it. One instruction is handled by One core, if someone needs multitasking or multiple instruction in parallel, minimum 2 cores are required.

We know when we insert or load data in a computer or server or in a node, it gets saved in hard disk. We won’t go in to details how it is stored on the hard disk. We can imagine when we write something on paper, consider the same way, data is saved in a hard disk.

Before we understand MDX, we must first reassemble our memories with respect to OLTP and OLAP. We know what these are, both are explained as separate topics in detail.

SQL (Structured Query Language) is for RELATIONAL DATABASES which is primarily used by OLTP systems, but it also used by almost all the datastores i.e., Data Warehouse, Data Lake, Data Vault, Business Vault etc.

MDX (Multidimensional eXpression) was invented by SQL Server engineers in 1997. MDX is the query language for OLAP DATABASES to extract multi-dimensional data from OLAP Cubes. Its syntax is just like XLS formula.

MDX is not a new language, it’s been there for decades but due to the ease of use of SQL language, it’s hardly used now a day. But experts are predicting the revival of MDX soon.

MDX Data Types

• Scalar can be number or string.
• Hierarchy is like dimension e.g., in time dimension, the hierarchy is year, month, week, day etc.
• Level is the level of Hierarchy of dimension like is year, month, or week etc.
• Member is the member of hierarchy like exact date or week or month etc.
• Tuples contain one or more members of a hierarchy.
• Set contains one more tuple.
• Other Data Types are the name of the fields/ columns in a multidimensional cube. It’s just like fields/ columns in a Data Warehouse or Data Mart.

Concept Drift is used in Data Science practice, and it is mostly unexpected behavior in data. One must have automated ways to keep tracking and identifying Concept Drifts as after Concept Drift, the model will totally not produce or predict the output which was planned. Wrong prediction means wrong decisions.

We all know what a Drift is. Drift is about shifting or moving from one point to another but not in a standard approach. This term is referred in Data Science practice to evaluate tested vs actual performance of Machine Learning (ML) Models.

Concept Drift is also known as class drift, real concept drift, or posterior probability shift. As we discussed in Data Drift, the data elements do not change rather data changes. But in Concept Drift, data elements themselves change.

For example, referring to the same example from Data Drift, model is trained on products but at the same time buying behavior of those products is dependent on external factors like weather. Let’s say, those products are Cold Drinks and it’s obvious in the region where it’s been sold, the weather is expected to be hot and based on the ML prediction, a production of 1M Cold Drinks are planned on weekly basis. Now, what if, due to any reason the weather sees a sudden change and for a week it rains, and the temperature drops unexpectedly, THIS IS CALLED CONCEPT DRIFT, then the whole model’s behavior will be impacted. Due to pleasant whether people will only consume 25% of 1M Cold Drinks which means the ML model must quickly change its training data and predict the production behavior.

Data Skew Issue is one of the main hidden culprits for query performance 😊. It is normally referred to as Skewed data distribution which is used in Data Science domain.

Have you ever noticed, your query is correct, your joins are correct, your all indexes are enabled, you have high data quality, you have data standardization in place but still your query performance is not as expected? You noticed when you loaded new datasets, queries were flying but as time passed, the performance has gone down. You have added more CPU, more RAM, more nodes but NO performance is not like newly added data. It’s because of the Data Skew Issue.

Data Skew Issue is about when data is not distribution evenly. Uneven distribution of data at cluster level causes this issue. It heavily degrades the performance of the queries especially those with joins.

We all have faced a situation where query run exceptionally good till 99% and then hangs for 3-4 hours right lolz yeah, I can understand your pain 😊. Let me tell you the reason, it’s because of Data Skew Issue.

ISO 55000, 55001, 550012 and 550013 are standards that provide guidance and best practices for managing assets effectively within an organization.

Let's break down these standards in simpler terms:

• ISO 55000 – A comprehensive series of international standards, plays a pivotal role in guiding organizations toward effective asset management practices. Within this framework, the concept of treating data as a valuable asset is gaining increasing prominence. ISO 55000 underscores the significance of data in asset management by emphasizing its role in decision-making processes and the overall lifecycle of assets. Organizations are encouraged to recognize data as a strategic asset, acknowledging its potential to enhance decision quality, mitigate risks, and optimize asset performance.
• ISO 55001 – a core component of the ISO 55000 series, provides a comprehensive framework for organizations to establish, implement, and continually improve their asset management systems. Within the context of ISO 55001, the recognition of data as a crucial asset is paramount. This standard encourages organizations to integrate data management practices into their asset management processes, acknowledging the pivotal role that data plays in decision-making and overall asset performance. ISO 55001 emphasizes the need for systematic approaches to managing data throughout the entire asset lifecycle, from acquisition to disposal. It underscores the importance of ensuring data accuracy, reliability, and relevance, as these factors significantly impact the effectiveness of asset management decisions. The standard promotes the establishment of robust data governance and information management processes, ensuring that data is treated with the same level of care and attention as other physical or financial assets. By incorporating data as a strategic asset within the ISO 55001 framework, organizations can harness the power of information to optimize asset performance, minimize risks, and enhance the overall value derived from their assets. This integration of data management principles into the asset management system aligns with contemporary best practices, acknowledging the critical role of data in achieving organizational objectives and sustaining long-term asset value.
• ISO 55012 (New) – The section on "Guidance on People Involvement and Competence" within the ISO 55000/1 standards serves as a comprehensive exploration of the human dimension in the realm of asset management. Recognizing the expanded scope of ISO 55000/1, which encompasses a broader understanding of asset management, this section specifically delves into the integration of data, information, and knowledge management within the human-centric aspects of the discipline. It provides detailed insights into how individuals within an organization, spanning various roles and levels, contribute to the effective implementation of asset management practices. This includes not only their technical competence but also their understanding and management of data, information, and knowledge as integral components supporting the overarching goals of asset management. By acknowledging the human-side dynamics, this guidance emphasizes the need for a skilled and knowledgeable workforce capable of leveraging data and information to make informed decisions, optimize asset performance, and ultimately contribute to the success of the asset management framework outlined in ISO 55000/1. In essence, it underscores the interconnectedness of human capabilities, data management, and achieving excellence in asset management practices.
• ISO 55013 (New)– The section titled "Guidance on the Management of Data Assets" represents a comprehensive and detailed examination of the pivotal role that data plays within the context of asset management. This segment is structured into primary clauses, each focusing on a distinct aspect of managing data as a valuable asset. The clauses include "Managing Asset Data," which outlines strategies for effectively handling data throughout its lifecycle; "Delivering Value from Asset Data," which underscores the importance of extracting meaningful insights and value from the data; "Identifying Data Assets," which guides organizations in recognizing and categorizing their data assets; "Managing Data Assets," providing insights into best practices for the ongoing management and maintenance of data; and "Governance," which emphasizes the need for a robust governance framework to ensure the integrity, security, and compliance of data assets.

ISO 11179 is an international standard that provides guidelines for the representation and management of metadata for describing data and information resources. In simpler terms, it offers a standardized way to organize and describe data, making it easier to understand, share, and manage.

Let's break down ISO 11179 in more detail:

As we have read in Metadata Management topic, Metadata is Data about Data. It describes the characteristics, properties, and context of data, helping users understand what the data is, how it was collected, and how it can be used.

ISO 11179 focuses on standardizing the way metadata is defined, organized, and managed. It provides a common framework for describing data elements, ensuring consistency and clarity in data descriptions. The standard helps organizations create a common language and structure for describing data, making it easier for different systems and people to understand and use the data.

MRKL stands for Modular Reasoning, Knowledge, and Language. This neuro-symbolic architecture is designed to address complex problems by combining elements of neural computation (specifically Large Language Models, or LLMs) with symbolic computation tools such as calculators. (ref: LLM is separate topic)

Neuro-Symbolic Architecture implies that the system combines aspects of neural computation (associated with LLMs) and symbolic computation. Neural computation involves processing information in a way inspired by the human brain, while symbolic computation involves manipulating symbols to perform operations.

LLMs (Large Language Models) are sophisticated artificial intelligence models trained on extensive text data. They surpass in recognizing and producing human-like dialect.

External Tools Like Calculators in addition to neural computation, the system incorporates external symbolic computation tools, specifically mentioning calculators. This suggests a hybrid approach where both neural and symbolic methods are employed.

A Watermark is a recognizable image or pattern embedded into a digital or physical document to identify its authenticity or to protect the document from unauthorized use. Watermarks are commonly used in various contexts, including photography, art, documents, and currency.

In the context of digital media, such as images or videos, a watermark is often a semi-transparent logo or text that is overlaid onto the content. This watermark serves several purposes:

• Copyright Protection: Content creators often use watermarks to assert their ownership of the material. This discourages unauthorized use and helps in identifying the original creator.
• Branding: Watermarks can also be used for branding purposes. Companies may add their logos or brand names as watermarks to promote brand recognition even when the content is shared online.
• Security: Watermarks can be used to secure confidential or sensitive documents. For example, companies might add watermarks to internal documents to discourage leaks or unauthorized distribution.
• Authentication: In the case of physical documents, such as banknotes or certificates, watermarks can be embedded during the papermaking process. These watermarks serve as a form of authentication to verify the document's legitimacy.