Knowledge Hub

“Semantic” is an ancient Greek word: σημαντικός sēmantikós, "significant" is the study of reference, meaning, or truth.

In Data World, Semantic layer is built for the sole purpose of supporting business users.

Semantic layer is to convert complex data models into simple and user-friendly interface for business users.

A semantic layer is the representation layer on top of databases, schema(s), tables and columns in such a way that Business users who have no technical know-how should be able to drag and drop required details on Dashboards. For drag and drop, first, one must assign business names to all databases, schema(s), tables and columns before exposing in BI tools. For example, there can be a column name as Date but the data in this column can be Date_Of_Birth or Transaction Date or Financial_Year_End_Data. Converting data models, column/ table/ schema/ database names as per business understanding is semantic process.

By now, we all know what Semantic Layer is, IT defines data elements for business ease. Then there came the era for Business folks defines data elements themselves based on their own understanding and now the latest approach is going towards Machine Learning Models define data element based on open-ended requirements.

A major difference between first two approaches vs Augmented Analytics is, first two were derived by humans either by IT or Business folks so there is always a factor of human era and at the same time human can be biased base on knowledge and information they possess about the business domain. On the other hand, Augmented Analytics is based on automated Machine Learning which learns the data elements while new data sets keep coming in.

The first approach was static as requirements were always coming from business and implemented by IT folks, so any change had a round robin effect.

Data Usability is the key to being a Data Driven Organization.

To better understand, let’s refer to the difference between Useful and Usable.

• Useful means e.g., a system is fulfilling the business requirements and able to perform all the operation tasks.
• Usable means e.g., how easy the system is for operational tasks i.e., how automated it is, which makes the operator’s life easy.

Let’s map it with Data World. Consider, the marketing team needs data from, let’s say, 5 source systems to send campaigns to premium bank customers. Now, dumping data from all 5 source systems at one centralized location like Data Lake might be USEFUL as all sort of data is available at one place but it’s not USABLE unless until it is not converted into a format which marketing team can easily use to send campaigns. In other words, unless data is not formatted in a Data Warehouse or a Data Mart or in a Semantic Layer, it might not be usable for relevant team.

Data Serving is also known as the Serving Layer and it has been there since its inception but as technologies have matured, many new channels have been introduced to support this layer.

‘Serving Layer is last component of the Lambda Architecture.’

Lambda Architecture has been explained in separate topic, so we won’t go into details of it but referring to it, as mentioned above, it is the last layer which is exposed to End Users for reporting purposes. This Layer is, if not all the time but majorly designed based on business requirements.

We all know OLTP systems are not meant to be exposed for reporting purposes. In the current era, for reporting, there are many data stores available i.e., Data Warehouse, OLAP, Data Lake, NoSQL, Semantic layers but ideally Semantic Layer should be treated as Serving Layer.

To be more specific

• If Serving Layer is designed for specific business requirement then Semantic Layer is last layer.
• If the Serving Layer is designed for generic use then it can be Data Warehouse, OLAP, Data Lake, NoSQL etc.

Consumption layer is confused with Serving Layer due to its name. Folks relates it that the layer which is used by Consumers to extract data. The Layer which Consumers use to extract information is Serving layer (explained separately).

Consumption Layer, as its name says it all, is a layer which consumes or takes-in data from source systems. In the current era, due to huge amounts of data coming in every second from sources like social media, CCTV, images, audios, sensors data etc. technology was required to consume data on a fast pace in MPP mode.

Tools like Hadoop, Object Storage and many RDBMS are giving MPP features now a days.

In Lambda Architecture, Serving Layer is the last layer, Consumption Layer is the layer which process data from source till Serving layer.

In the image, Batch and Speed Layers are Consumption Layers. In an OLAP system, the consumption layer is Data Warehouse and Data Mart, considering there are OLAP Cubes and Semantic layers but in a Data Warehouse, consumption layer is Landing and Staging zone.

Data Warehouse

A data warehouse is a type of data platform that is designed to store and analyze structured data from various sources, providing a central repository for business intelligence (BI) and analytics applications. The data stored in a data warehouse is typically historical, integrated, and cleansed, and can be used to support decision-making, reporting, and analysis at all levels of an organization. The data stored in a data warehouse is typically structured and organized into tables and is optimized for efficient querying and analysis.

Data Lake

A data lake is a centralized repository that allows you to store all your structured, semi and unstructured data at any scale. The term was coined to represent the idea of a system where data flows in from various sources and resides in its most "natural" state, just like water in a lake. These data can range from raw, unprocessed files to transformed, structured records.

Slowly Changing Dimensions is also known as SCD and is used in Data Warehouse concept. Folks in Data World are very much aware of this term.

Before we understand SCD, one must understand Fact and Dimensions. In dimensional Data Warehouse or in a Data Mart (explained in separate topic), there are two types of schema(s) i.e., Star and Snowflake Schema(s). Both are build based on Facts and Dimensions.

• Facts: This table keeps the transaction’s lowest level of data which can be numeric or non-numeric like sales numbers, number of clicks on social media portal, number of searches in google etc. It’s mainly loaded in append mode, means rows always get inserted. This table is a very fast-moving table, and rows keep coming in. There are hardly any organizations which update and delete data from Fact Tables.
• Dimension: This table on the other hand keeps data element by which Fact can be analyzed e.g., sale by time, sales by region, sales by the product etc. So, Time, Regions, Products etc., are dimensions. In contrast with Fact table, frequency of rows coming in these tables are very low. For Example, once we have loaded regions then when often regions will change 😊. Once Time is loaded, it will never change. At the same time, Products definitions can change. A company today has 100 products, in future it can have more. Then attributes of one product can change in future a well, so there are three ways to keep the history of changed in Dimensions which are handled by principles managed by SLOWLY CHANGING DIMENSIONS (SCD).

Materialized View is a database object that stores the output of a query. Unlike a regular view, which is a virtual table derived from the result of a SELECT query, a materialized view physically stores the data in a structured form. This precomputed data can be periodically refreshed to reflect changes in the underlying tables.

It is a snapshot of the data at the time the materialized view was last refreshed. You create a materialized view by defining the structure of the view and specifying the query that populates it.

CREATE MATERIALIZED VIEW EmployeeProfile

AS

SELECT Employee.*, Department.*

FROM Employee, Department

WHERE Employee.DeptID = Department.DeptID

In this example, we create a materialized view named " EmployeeProfile " that summarizes Employees Details based on Employee and Department tables.

Unlike regular views, materialized views store the data physically. The result set of the query materialized, meaning it's stored in a table-like structure within the database.

Yes, you are right, in Data World, Staging Layer is one of the most important zones then why to write about it. Trust me it's been underutilized in most implementations.

Let's decode it.....

The Staging Layer in data architecture plays a crucial role in the data processing pipeline, providing a temporary and structured area for incoming data before it is integrated into the data warehouse or other storage systems.

Many a times Staging Layer is confused with Landing Layer. Yeah trust me, I have seen people doing nothing in Staging Layer except minor things. Please note, especially in the era of cloud, if one doesn’t properly utilize Staging Layer, the consumption cost from Staging Layer to Transformed Layer will be significantly higher.

The importance of the staging layer lies in several key aspects:

• Data Quality and Validation: The staging layer acts as a checkpoint for data quality and validation. Before data is moved to the main storage or processing area, it undergoes validation checks to ensure it meets quality standards. This helps identify and address issues such as missing values, formatting errors, or inconsistencies early in the data pipeline.
• Data Enrichment: Staging allows for data enrichment processes to take place before the data reaches its final destination. This step involves cleaning, enhancing the data, making it more suitable for analysis and reporting. It ensures that the data conforms to the desired format and structure.
• Incremental Loading: The staging layer facilitates incremental loading of data, allowing for efficient updates and additions. Instead of reloading the entire dataset, only new or modified data needs to be processed, reducing the computational load and improving overall system performance.
• Data Auditing and Lineage: Staging provides a historical record of the ingested data, enabling auditing and tracing of changes over time. This is essential for tracking the lineage of data, understanding its origin, and ensuring compliance with data governance and regulatory requirements.
• Fault Tolerance and Recovery: Staging serves as a buffer against data processing failures. If an error occurs during data integration or transformation, the staging layer allows for easier recovery without affecting the main data storage. It provides a level of fault tolerance and helps maintain the integrity of the overall data processing pipeline.
• Isolation of Source and Destination Systems: Staging separates the source systems (where data originates) from the destination systems (where data is stored for analysis). This isolation ensures that any changes or issues in the source systems do not immediately impact the data warehouse or analytics environment, providing a layer of protection.
• Performance Optimization: By pre-processing and validating data in the staging layer, the main data storage systems are relieved of some of the computational overhead. This optimization contributes to improved performance and responsiveness in data processing.

Over the years, our approach to measuring marketing performance has drastically evolved and continues to change with advancing technology and evolving data policies. The days of rigidly assigning budgets to different marketing channels and relying solely on P&L statements to gauge the success of our efforts are behind us.

Assessing the performance of our ads involves understanding the correlation between our marketing spend and revenue generated. With data sources and clear models, especially when running campaigns across various channels simultaneously (e.g., Facebook, Google, Out Of Home, TV), the key question remains:

Which channels and touchpoints contribute significantly to the final sale?

To evaluate that, we employ distinct data models in marketing, each carrying its own set of advantages and limitations. These models must be tailored to fit each use case and align with how people interact with our business: