Knowledge Hub

EDI, or Electronic Data Interchange has been introduced since the seventies. The main use case addressed by EDI is to automate data exchange between trading partners (Customers and Suppliers) for business documents, such as purchase orders and invoices as well as for payments and shipment information.

Here is a simple EDI use case in which Smart Electronics (a fictitious Manufacturing company) uses EDI to send Purchase Orders to Supplier S, and Supplier S sends corresponding invoices to Smart Electronics.

EDI follows specific message standards and formats like EDIFACT or X12 which are the main EDI message standards.

Other EDI standards have been developed for specific purposes or industries, such as:

• ODETTE: used in automotive industry in European countries
• HL7: used for healthcare data.
• IATA PATIS: used to exchange passenger and airport data, and is maintained by the IATA (the International Air Transport Association)
• SWIFT: used for transaction and message exchanges in the financial and banking sectors.

Socratic questioning is a form of inquiry and discussion named after the ancient Greek philosopher Socrates. It involves a method of cooperative argumentative dialogue to stimulate critical thinking and illuminate ideas. Socratic questions are designed to encourage individuals to think deeply, consider various perspectives, and arrive at their own conclusions. The goal is not to provide answers but to facilitate a process of self-discovery and learning.

When applying Socratic questioning to data solutions, the goal is to stimulate critical thinking, uncover assumptions, and encourage a deeper understanding of the data-related challenges or opportunities. Here's how Socratic questioning can be applied in the context of data solutions:

• Clarifying Questions: Can you clarify the specific data issue you're trying to address to ensure a clear understanding of the problem or goal in the context of data solutions.
• Probing Assumptions: What assumptions are you making about the quality of the data to uncover any preconceived notions or assumptions that might impact the approach to handling data.
• Exploring Evidence and Reasons: What evidence supports the decision to use a particular data processing method to understand the rationale behind the chosen data processing techniques or tools.
• Considering Alternative Perspectives: How might a data scientist approach this problem differently than a business analyst to encourage thinking about different viewpoints and approaches within the realm of data solutions.
• Examining Implications and Consequences: What are the potential consequences of using incomplete or inaccurate data in decision-making to explore the downstream effects of data-related decisions and their impact on the organization?

Over the years, our approach to measuring marketing performance has drastically evolved and continues to change with advancing technology and evolving data policies. The days of rigidly assigning budgets to different marketing channels and relying solely on P&L statements to gauge the success of our efforts are behind us.

Assessing the performance of our ads involves understanding the correlation between our marketing spend and revenue generated. With data sources and clear models, especially when running campaigns across various channels simultaneously (e.g., Facebook, Google, Out Of Home, TV), the key question remains:

Which channels and touchpoints contribute significantly to the final sale?

To evaluate that, we employ distinct data models in marketing, each carrying its own set of advantages and limitations. These models must be tailored to fit each use case and align with how people interact with our business:

Success in the dynamic landscape of marketing depends on engaging the right audience effectively. However, not all audiences respond the same to advertisements.

At the core of this lies a pivotal question:

Would customers buy without the corresponding paid advertising?
This question is at the core of incrementality in marketing, revealing three different ways people react to ads.

Understanding Audience Behavior

This basic exploration breaks down how audiences react into three different groups:

1. Non-responsive Audience: This group is unaffected by ads no matter how much they see them. They don't feel like engaging or buying because of ads.
2. Pre-existing Demand Audience: People in this group were already planning to buy, ads or not. Ads might change when they buy, but they won't change their mind about buying.
3. Incremental Audience: Importantly, this group includes people who are directly affected by ads. These potential customers engage or buy only because they saw ads, showing how much ads affect their behavior.

The role of ontology in data integration and harmonization is to describe structure and relations. Ontologies enable a coherent structure for data across systems, functions, and domains. As the cost of maintaining and point-to-point integrations is growing along with the cost for running specific data pipelines there is a need for a new approach. By having an ontological approach to an organization’s data-driven strategy cost control will be easier to achieve.

The Concept of Ontology in Data Management

Ontologies in data management involve the use of a shared vocabulary and a set of relationships and rules to model a domain. They provide a common language for data and thus enable different systems and stakeholders to 'speak' the same language.

Data Integration Challenges

• Disparate Data Sources: Organizations often deal with data from various sources each with their unique format and structure.
• Semantic Discrepancies: Different interpretations of data across systems can lead to inconsistencies and misunderstandings.

To succeed with metadata management, you will need an Operational Reference Model. It is the way to describe the structures and relationships of your business metadata.

The Role and relationship of an Operational Reference Model

Conceptual models, logical models, physical models and canonical models have a specific purpose which is either communication and understanding, or representation of an individual system. Operational Reference Models come into play when organizations require an operational layer which bridges multiple systems and functions and provides a common point of reference or shared language for the enterprise. The Operational Reference Model is similar. to an anti-corruption layer and a semantic layer in its purpose but with an important difference, it's owned and governed by your subject matter experts.

Operational reference model governance

An operational reference model is at its core a set of metadata structured with the help of taxonomies and ontologies. Each part, each domain, is governed by the subject matter experts in the organization in accordance with a federated governance approach.

In typical SQL world (Structured Query Language), usually managing and manipulating data efficiently is very crucial for its performance, database lifecycle or DWH scalability. Temporary tables and materialized views are two powerful constructs (objects) that help in achieving both objectives simultaneously. Even though each serves a unique purpose within data operations and data streams. Let´s look at both little closer.

Temporary Tables

We can define them as typical tables which are created within a database session and are only available during the lifetime of that particular session. Lifetime during session is critical as they are designed to store intermediate results, results of window functions for complex calculations or data processing tasks and massive data manipulations.

Temporary tables are essentially useful in scenarios where data needs to be temporarily held and manipulated without affecting the main database tables. Here comes why it is called “temporary”. After your session ends, the temporary table is automatically discarded - same as you run command DROP TABLE, after query results are taken into another step. This way processing ensures no permanent database storage is consumed. You can imagine storing all 245 steps and their output tables consisting of 6 billion rows and 700 columns each day to have versions of each step.

During the years of industrialization and first process automations, there was the first batch of international standards which has become the core of ISO “framework”. When data stormed through various industries, ISO/IEC 27001 was necessary to be defined. It is an international standard specifying the requirements for establishing, implementing, maintaining, and continually improving an information security management system (ISMS). From a data perspective, this standard is core, essentially pivotal to ensure the confidentiality, integrity, and availability (CIA) of data. It consists of 3 main sections, which encompasses policies, processes, and technical measures to manage information security risks.

There are 6 Key Components when dealing with data and data management.

• The foundational aspect for ISO/IEC 27001 is the systematic assessment of information security risks in your own data. Risk assessment includes identifying potential threats to data, assessing vulnerabilities, and risk impacts’ evaluation. The outcome guides the prioritization of security processes, monitoring and controls.
• Managing risk effectively, data must be classified according to its sensitivity and importance to the organization and its departments. Such classification influences how data is handled, stored, and transmitted, ensuring that more sensitive data receives higher levels of protection. The processing phase is called data classification.
• ISO/IEC 27001 highlights massive importance for limiting access to data based on the principle of least privilege. Access control lineage and its measures ensure that only authorized individuals can access or modify data, reducing the risk of unauthorized disclosure, alteration, partial loss, or total destruction.
• Encrypting data (both at rest and in transit], is a very common control in data protection process for keeping confidentiality and integrity. Encryption makes data unreadable to unauthorized parties, safeguarding it against spying or undesired interfering.
• Next component the ISO standard 27001 requires for all organizations to have procedures for managing information security incidents. Incident Management includes detecting, reporting, and investigating each incident to mitigate their impact on data and prevent recurrence.
• Last but not least ISO/IEC 27001 brought into place Business Continuity Management to keep development of any business continuity plan which includes data recovery procedures. “Continuity” ensures that critical data remains available or can be restored in the event of a disruption. Under “event” you can imagine a typical cyber-attack or natural disaster.

Let’s start by breaking down each part of this acronym based on the context of its usage.

• Information – In this case the Information is related to the usage of the related data stored in a database or a data platform.
• Lifecycle – The standard definition for life cycle is the different stages of life for a living thing. In this case the living thing is the Data. The Lifecycle for data is classified based on its usage as: - ‘HOT’, ‘WARM’, ‘COLD’ and that also could be extended to ‘DEAD’ for deleted.
• Management – The process to facilitate the movement of the data related to the information over its Lifecycle.

So, using the context above, Information Lifecycle Management (ILM) is the process to optimize data storage by automatically tiering data to different locations based on its age, access frequency, and required read/write speeds. This approach minimizes storage costs by placing frequently accessed data in high-performance tiers and infrequently accessed or archived data in lower-cost tiers. ILM can also simplify data archiving and deletion according to data retention policies.

Reference Architecture is foundational to designing any complex system or solution. It provides a high-level diagram that represents the structural pattern of the solution, establishing a consistent framework that guides the design and implementation of similar projects across an organization. Unlike solution architecture, which is customized for each project, reference architecture offers a generalized template that aligns with established best practices, making it broadly applicable and easily understandable for both technical and non-technical stakeholders.

Purpose and Benefits

The main objective of a reference architecture is to set a standardized approach for designing solutions within an organization. This ensures consistency, reduces redundancy, and accelerates solution delivery by offering a clear blueprint. With this architecture, teams can quickly understand the fundamental elements of the solution, enabling efficient alignment with organizational goals. Additionally, reference architectures aid in risk management by promoting tried-and-tested design patterns and technologies, minimizing potential errors in execution.