Knowledge Hub

Geocoding has become of the most demanding data element in current era where data is generated everywhere and by everyone.

‘Geocoding refers to the geographic location.’

Geocoding is about the name, street, city, regions etc., rather exact location on Earth. The exact data elements representing Geocoding are Longitude and Latitude, also known as Lat and Long. Using Lat and Long, one can reach the exact location, anywhere in the World.

In the current era social media, mobile applications, sensors, satellite scans, CCTV, online video(s), audio(s) etc., all have become a norm. Millions of users are using mobile on a daily basis, everywhere, all the time around the globe. Location information has become one of the most important pieces of information for everyone especially consumer details.

For example, we all know Amazon, it’s one the most used online stores around the globe which is used by millions of people to buy and sell products, 24/7. If Amazon does not know where the buyer is from, and what is the location where most buyers of a particular product are, then it can’t plan its marketing campaigns.

Data Matching and Linking is for multiple purposes across systems.

Data Matching and Linking is about matching and linking datasets across the systems to find out which is the best record to link with. Generating Golden Record for MDM (explained separately) can be one of the use cases. Additionally, we are also referring to when the same nature of data is coming from multiple source systems and there is a need to find the best possible record.

Please note, we are not talking about manual matching and linking, we are talking about complex software(s) using complex algorithms to find the right data.

For example, customer onboard process can be done Online or Customer Service Centre or at a Branch. If the same customer gets registered 2 to 3 times via different channels, then one must use Data Matching and Linking process to identify customer with high weightage.

Types

• Referential is where exact fields are matched and linked like National ID, Social Security Number, Password numbers etc.
• Probabilistic is where multiple fields are used like name, date of birth, parent’s names etc.
• ML Association is where machine learning searches for matching data element to link.

Data Serving is also known as the Serving Layer and it has been there since its inception but as technologies have matured, many new channels have been introduced to support this layer.

‘Serving Layer is last component of the Lambda Architecture.’

Lambda Architecture has been explained in separate topic, so we won’t go into details of it but referring to it, as mentioned above, it is the last layer which is exposed to End Users for reporting purposes. This Layer is, if not all the time but majorly designed based on business requirements.

We all know OLTP systems are not meant to be exposed for reporting purposes. In the current era, for reporting, there are many data stores available i.e., Data Warehouse, OLAP, Data Lake, NoSQL, Semantic layers but ideally Semantic Layer should be treated as Serving Layer.

To be more specific

• If Serving Layer is designed for specific business requirement then Semantic Layer is last layer.
• If the Serving Layer is designed for generic use then it can be Data Warehouse, OLAP, Data Lake, NoSQL etc.

Consumption layer is confused with Serving Layer due to its name. Folks relates it that the layer which is used by Consumers to extract data. The Layer which Consumers use to extract information is Serving layer (explained separately).

Consumption Layer, as its name says it all, is a layer which consumes or takes-in data from source systems. In the current era, due to huge amounts of data coming in every second from sources like social media, CCTV, images, audios, sensors data etc. technology was required to consume data on a fast pace in MPP mode.

Tools like Hadoop, Object Storage and many RDBMS are giving MPP features now a days.

In Lambda Architecture, Serving Layer is the last layer, Consumption Layer is the layer which process data from source till Serving layer.

In the image, Batch and Speed Layers are Consumption Layers. In an OLAP system, the consumption layer is Data Warehouse and Data Mart, considering there are OLAP Cubes and Semantic layers but in a Data Warehouse, consumption layer is Landing and Staging zone.

We all have heard about MPP (Massive Parallel Processing). In the current era, consumption and serving layer without MPP capability cannot survive.

MPP is mostly used to read and write (mostly insert) in parallel where more than 1 processor is involved.

In MPP, multiple processers are used to process queries in parallel which might be running in single node or across multiple nodes, in same location or multiple locations.

In a non-MPP mode when a query is executed its different pieces of code are executed one by one in a sequence. But in MPP mode when a query is executed its different pieces of code are executed in parallel on all the processors in one go. This increases the execution of queries 100 times faster than compared to one-by-one execution.

In the current era where data coming from social-media, audios, videos etc., are coming in such a velocity/ pace/ speed that without MPP, it will not be possible to write, process or read data in timely manner.

Canned Data is something which is readily available at one click. A dataset prepared based on business requirements i.e., if required in pre-aggregated format, in one accessible format.

The aim of canned data is to have a dataset that needs no more cleansing, no more wrangling, no more reconciliation etc. Aim is, User to just click it and data should be served.

Imagine Canned food, which are prepared and seals to be used anytime, anywhere, by anyone. Most of the canned food is ready to eat. Same is with Canned data, ready to use. OLAP Cubes are a perfect example of canned data where pre-calculated and pre-aggregated data is stored in a multi-dimensional table, ready to be consumed anytime.

‘Canned data can be refreshed with latest data using same existing data element.’

Canned Reports

Just like Canned Data, Canned Reports has the same approach where static designed reports are built for non-tech savvy stakeholders, who just want to click once and see the expected required output.

Adhoc Reports

Whereas Adhoc reports are for tech savvy users who know how to play with the BI tools and who also have business knowledge of data element shared in the tool. Using Adhoc reporting, users can build their own reports, dashboards by drag and drop exposed tables and columns. Users can slice and dice the reports. Users can drill-in and drill-out.

Tools like

• Oracle Forms, SSRS etc. are considered as Canned Reports generators.
• Tableau, Power BI, Qlik etc. are considered as Adhoc Reports generators.
• OBIEE, Cognos etc. are considered as Semi Canned and Semi Adhoc Reports generators.

Before we understand MDX, we must first reassemble our memories with respect to OLTP and OLAP. We know what these are, both are explained as separate topics in detail.

SQL (Structured Query Language) is for RELATIONAL DATABASES which is primarily used by OLTP systems, but it also used by almost all the datastores i.e., Data Warehouse, Data Lake, Data Vault, Business Vault etc.

MDX (Multidimensional eXpression) was invented by SQL Server engineers in 1997. MDX is the query language for OLAP DATABASES to extract multi-dimensional data from OLAP Cubes. Its syntax is just like XLS formula.

MDX is not a new language, it’s been there for decades but due to the ease of use of SQL language, it’s hardly used now a day. But experts are predicting the revival of MDX soon.

MDX Data Types

• Scalar can be number or string.
• Hierarchy is like dimension e.g., in time dimension, the hierarchy is year, month, week, day etc.
• Level is the level of Hierarchy of dimension like is year, month, or week etc.
• Member is the member of hierarchy like exact date or week or month etc.
• Tuples contain one or more members of a hierarchy.
• Set contains one more tuple.
• Other Data Types are the name of the fields/ columns in a multidimensional cube. It’s just like fields/ columns in a Data Warehouse or Data Mart.

Data Drift concept is used in Data Science practice, and it is one of the most expected behaviors in every data modelling. One must have automated ways to keep tracking and identifying Data Drifts as after Data Drift, there are high chances the model will not produce or predict the expected accuracy which was planned. Wrong prediction means wrong decisions.

We all know what a Drift is. Drift is about shifting or moving from one point to another but not in a standard approach. This term is mostly referred in Data Science practice, to evaluate trained vs actual., performances of Machine Learning (ML) Models.

Data Drift is also known as covariate shift, virtual drift, or virtual concept drift. When we create and run a ML model, just like any traditional query, it inputs a dataset and outputs a dataset. In ` ML modelling, we select certain data elements to build its logic. Data Drift is about change of data within same select data element.

Concept Drift is used in Data Science practice, and it is mostly unexpected behavior in data. One must have automated ways to keep tracking and identifying Concept Drifts as after Concept Drift, the model will totally not produce or predict the output which was planned. Wrong prediction means wrong decisions.

We all know what a Drift is. Drift is about shifting or moving from one point to another but not in a standard approach. This term is referred in Data Science practice to evaluate tested vs actual performance of Machine Learning (ML) Models.

Concept Drift is also known as class drift, real concept drift, or posterior probability shift. As we discussed in Data Drift, the data elements do not change rather data changes. But in Concept Drift, data elements themselves change.

For example, referring to the same example from Data Drift, model is trained on products but at the same time buying behavior of those products is dependent on external factors like weather. Let’s say, those products are Cold Drinks and it’s obvious in the region where it’s been sold, the weather is expected to be hot and based on the ML prediction, a production of 1M Cold Drinks are planned on weekly basis. Now, what if, due to any reason the weather sees a sudden change and for a week it rains, and the temperature drops unexpectedly, THIS IS CALLED CONCEPT DRIFT, then the whole model’s behavior will be impacted. Due to pleasant whether people will only consume 25% of 1M Cold Drinks which means the ML model must quickly change its training data and predict the production behavior.