Knowledge Hub

Before we understand MDX, we must first reassemble our memories with respect to OLTP and OLAP. We know what these are, both are explained as separate topics in detail.

SQL (Structured Query Language) is for RELATIONAL DATABASES which is primarily used by OLTP systems, but it also used by almost all the datastores i.e., Data Warehouse, Data Lake, Data Vault, Business Vault etc.

MDX (Multidimensional eXpression) was invented by SQL Server engineers in 1997. MDX is the query language for OLAP DATABASES to extract multi-dimensional data from OLAP Cubes. Its syntax is just like XLS formula.

MDX is not a new language, it’s been there for decades but due to the ease of use of SQL language, it’s hardly used now a day. But experts are predicting the revival of MDX soon.

MDX Data Types

• Scalar can be number or string.
• Hierarchy is like dimension e.g., in time dimension, the hierarchy is year, month, week, day etc.
• Level is the level of Hierarchy of dimension like is year, month, or week etc.
• Member is the member of hierarchy like exact date or week or month etc.
• Tuples contain one or more members of a hierarchy.
• Set contains one more tuple.
• Other Data Types are the name of the fields/ columns in a multidimensional cube. It’s just like fields/ columns in a Data Warehouse or Data Mart.

Concept Drift is used in Data Science practice, and it is mostly unexpected behavior in data. One must have automated ways to keep tracking and identifying Concept Drifts as after Concept Drift, the model will totally not produce or predict the output which was planned. Wrong prediction means wrong decisions.

We all know what a Drift is. Drift is about shifting or moving from one point to another but not in a standard approach. This term is referred in Data Science practice to evaluate tested vs actual performance of Machine Learning (ML) Models.

Concept Drift is also known as class drift, real concept drift, or posterior probability shift. As we discussed in Data Drift, the data elements do not change rather data changes. But in Concept Drift, data elements themselves change.

For example, referring to the same example from Data Drift, model is trained on products but at the same time buying behavior of those products is dependent on external factors like weather. Let’s say, those products are Cold Drinks and it’s obvious in the region where it’s been sold, the weather is expected to be hot and based on the ML prediction, a production of 1M Cold Drinks are planned on weekly basis. Now, what if, due to any reason the weather sees a sudden change and for a week it rains, and the temperature drops unexpectedly, THIS IS CALLED CONCEPT DRIFT, then the whole model’s behavior will be impacted. Due to pleasant whether people will only consume 25% of 1M Cold Drinks which means the ML model must quickly change its training data and predict the production behavior.

Data Skew Issue is one of the main hidden culprits for query performance 😊. It is normally referred to as Skewed data distribution which is used in Data Science domain.

Have you ever noticed, your query is correct, your joins are correct, your all indexes are enabled, you have high data quality, you have data standardization in place but still your query performance is not as expected? You noticed when you loaded new datasets, queries were flying but as time passed, the performance has gone down. You have added more CPU, more RAM, more nodes but NO performance is not like newly added data. It’s because of the Data Skew Issue.

Data Skew Issue is about when data is not distribution evenly. Uneven distribution of data at cluster level causes this issue. It heavily degrades the performance of the queries especially those with joins.

We all have faced a situation where query run exceptionally good till 99% and then hangs for 3-4 hours right lolz yeah, I can understand your pain 😊. Let me tell you the reason, it’s because of Data Skew Issue.

ISO 55000, 55001, 550012 and 550013 are standards that provide guidance and best practices for managing assets effectively within an organization.

Let's break down these standards in simpler terms:

• ISO 55000 – A comprehensive series of international standards, plays a pivotal role in guiding organizations toward effective asset management practices. Within this framework, the concept of treating data as a valuable asset is gaining increasing prominence. ISO 55000 underscores the significance of data in asset management by emphasizing its role in decision-making processes and the overall lifecycle of assets. Organizations are encouraged to recognize data as a strategic asset, acknowledging its potential to enhance decision quality, mitigate risks, and optimize asset performance.
• ISO 55001 – a core component of the ISO 55000 series, provides a comprehensive framework for organizations to establish, implement, and continually improve their asset management systems. Within the context of ISO 55001, the recognition of data as a crucial asset is paramount. This standard encourages organizations to integrate data management practices into their asset management processes, acknowledging the pivotal role that data plays in decision-making and overall asset performance. ISO 55001 emphasizes the need for systematic approaches to managing data throughout the entire asset lifecycle, from acquisition to disposal. It underscores the importance of ensuring data accuracy, reliability, and relevance, as these factors significantly impact the effectiveness of asset management decisions. The standard promotes the establishment of robust data governance and information management processes, ensuring that data is treated with the same level of care and attention as other physical or financial assets. By incorporating data as a strategic asset within the ISO 55001 framework, organizations can harness the power of information to optimize asset performance, minimize risks, and enhance the overall value derived from their assets. This integration of data management principles into the asset management system aligns with contemporary best practices, acknowledging the critical role of data in achieving organizational objectives and sustaining long-term asset value.
• ISO 55012 (New) – The section on "Guidance on People Involvement and Competence" within the ISO 55000/1 standards serves as a comprehensive exploration of the human dimension in the realm of asset management. Recognizing the expanded scope of ISO 55000/1, which encompasses a broader understanding of asset management, this section specifically delves into the integration of data, information, and knowledge management within the human-centric aspects of the discipline. It provides detailed insights into how individuals within an organization, spanning various roles and levels, contribute to the effective implementation of asset management practices. This includes not only their technical competence but also their understanding and management of data, information, and knowledge as integral components supporting the overarching goals of asset management. By acknowledging the human-side dynamics, this guidance emphasizes the need for a skilled and knowledgeable workforce capable of leveraging data and information to make informed decisions, optimize asset performance, and ultimately contribute to the success of the asset management framework outlined in ISO 55000/1. In essence, it underscores the interconnectedness of human capabilities, data management, and achieving excellence in asset management practices.
• ISO 55013 (New)– The section titled "Guidance on the Management of Data Assets" represents a comprehensive and detailed examination of the pivotal role that data plays within the context of asset management. This segment is structured into primary clauses, each focusing on a distinct aspect of managing data as a valuable asset. The clauses include "Managing Asset Data," which outlines strategies for effectively handling data throughout its lifecycle; "Delivering Value from Asset Data," which underscores the importance of extracting meaningful insights and value from the data; "Identifying Data Assets," which guides organizations in recognizing and categorizing their data assets; "Managing Data Assets," providing insights into best practices for the ongoing management and maintenance of data; and "Governance," which emphasizes the need for a robust governance framework to ensure the integrity, security, and compliance of data assets.

ISO 11179 is an international standard that provides guidelines for the representation and management of metadata for describing data and information resources. In simpler terms, it offers a standardized way to organize and describe data, making it easier to understand, share, and manage.

Let's break down ISO 11179 in more detail:

As we have read in Metadata Management topic, Metadata is Data about Data. It describes the characteristics, properties, and context of data, helping users understand what the data is, how it was collected, and how it can be used.

ISO 11179 focuses on standardizing the way metadata is defined, organized, and managed. It provides a common framework for describing data elements, ensuring consistency and clarity in data descriptions. The standard helps organizations create a common language and structure for describing data, making it easier for different systems and people to understand and use the data.

MRKL stands for Modular Reasoning, Knowledge, and Language. This neuro-symbolic architecture is designed to address complex problems by combining elements of neural computation (specifically Large Language Models, or LLMs) with symbolic computation tools such as calculators. (ref: LLM is separate topic)

Neuro-Symbolic Architecture implies that the system combines aspects of neural computation (associated with LLMs) and symbolic computation. Neural computation involves processing information in a way inspired by the human brain, while symbolic computation involves manipulating symbols to perform operations.

LLMs (Large Language Models) are sophisticated artificial intelligence models trained on extensive text data. They surpass in recognizing and producing human-like dialect.

External Tools Like Calculators in addition to neural computation, the system incorporates external symbolic computation tools, specifically mentioning calculators. This suggests a hybrid approach where both neural and symbolic methods are employed.

A Watermark is a recognizable image or pattern embedded into a digital or physical document to identify its authenticity or to protect the document from unauthorized use. Watermarks are commonly used in various contexts, including photography, art, documents, and currency.

In the context of digital media, such as images or videos, a watermark is often a semi-transparent logo or text that is overlaid onto the content. This watermark serves several purposes:

• Copyright Protection: Content creators often use watermarks to assert their ownership of the material. This discourages unauthorized use and helps in identifying the original creator.
• Branding: Watermarks can also be used for branding purposes. Companies may add their logos or brand names as watermarks to promote brand recognition even when the content is shared online.
• Security: Watermarks can be used to secure confidential or sensitive documents. For example, companies might add watermarks to internal documents to discourage leaks or unauthorized distribution.
• Authentication: In the case of physical documents, such as banknotes or certificates, watermarks can be embedded during the papermaking process. These watermarks serve as a form of authentication to verify the document's legitimacy.

The role of ontology in data integration and harmonization is to describe structure and relations. Ontologies enable a coherent structure for data across systems, functions, and domains. As the cost of maintaining and point-to-point integrations is growing along with the cost for running specific data pipelines there is a need for a new approach. By having an ontological approach to an organization’s data-driven strategy cost control will be easier to achieve.

The Concept of Ontology in Data Management

Ontologies in data management involve the use of a shared vocabulary and a set of relationships and rules to model a domain. They provide a common language for data and thus enable different systems and stakeholders to 'speak' the same language.

Data Integration Challenges

• Disparate Data Sources: Organizations often deal with data from various sources each with their unique format and structure.
• Semantic Discrepancies: Different interpretations of data across systems can lead to inconsistencies and misunderstandings.

To succeed with metadata management, you will need an Operational Reference Model. It is the way to describe the structures and relationships of your business metadata.

The Role and relationship of an Operational Reference Model

Conceptual models, logical models, physical models and canonical models have a specific purpose which is either communication and understanding, or representation of an individual system. Operational Reference Models come into play when organizations require an operational layer which bridges multiple systems and functions and provides a common point of reference or shared language for the enterprise. The Operational Reference Model is similar. to an anti-corruption layer and a semantic layer in its purpose but with an important difference, it's owned and governed by your subject matter experts.

Operational reference model governance

An operational reference model is at its core a set of metadata structured with the help of taxonomies and ontologies. Each part, each domain, is governed by the subject matter experts in the organization in accordance with a federated governance approach.

A Unified Approach to Data Ecosystem Management

The concept of Data Platform Gravity revolves around a holistic framework where every aspect of data handling, from ingestion to transformation, analytics, and deployment, is achieved within a single, unified platform. Inspired by the Data Fabric paradigm, it advocates for seamless operations across the entire data lifecycle, reducing complexities, minimizing data silos, and empowering organizations to derive value efficiently and effectively.

Understanding Data Platform Gravity

In a traditional data management ecosystem, disparate tools and technologies often create bottlenecks. A single project might involve different platforms for:

• Extracting and loading data from sources,
• Transforming and enriching data,
• Performing analytics and data science experiments,
• Visualizing results in dashboards or reports, and
• Sharing data insights with external systems or stakeholders.