Knowledge Hub

Data Skew Issue is one of the main hidden culprits for query performance 😊. It is normally referred to as Skewed data distribution which is used in Data Science domain.

Have you ever noticed, your query is correct, your joins are correct, your all indexes are enabled, you have high data quality, you have data standardization in place but still your query performance is not as expected? You noticed when you loaded new datasets, queries were flying but as time passed, the performance has gone down. You have added more CPU, more RAM, more nodes but NO performance is not like newly added data. It’s because of the Data Skew Issue.

Data Skew Issue is about when data is not distribution evenly. Uneven distribution of data at cluster level causes this issue. It heavily degrades the performance of the queries especially those with joins.

We all have faced a situation where query run exceptionally good till 99% and then hangs for 3-4 hours right lolz yeah, I can understand your pain 😊. Let me tell you the reason, it’s because of Data Skew Issue.

Coupling is a parameter dependency approach used cross software or cross systems.

This approach is used when there are more than one software or system is in the picture to deliver the result.

Data Coupling is about when data elements are used, to kick-off the next step in a different system. For example, in data pipeline, we are pulling data from a hospital management system e.g., when a baby is born let’s have a data dependency like, the moment the baby is registered in system, pull its data. So, it’s a cross-system data dependency or we called Data Coupling.

The disadvantage of Data Coupling approach is, if something goes wrong in the first system, dependent systems cannot move.

Let me share another old example. Old data pipeline folks will remember this one i.e., when we used to load files from another system, we had to wait till the time whole file movement is completed. The challenge was how to make sure when to start pulling like only once file is completely copied.

Data Imputation is majorly used in Data Science domain.

‘Imputation means, fill missing data.’

Data Imputation normally comes into play so business rules can be imposed on all data rather on subsets of data. Before Machine Learning Model runs, data imputation is done by either missing data is filled up or the whole row is removed.

Missing value can create many kinds of issues e.g., during categorizing, classifying, clustering etc., if data is missing then results will not be 100%.

Generally, in numeric datasets, missing values are replaced with MEAN values. Having said that, there many ways to deal with missing data.

Few Imputations Techniques

• The first technique is where data is missing at random basis but before using it, it is completely removed from the dataset.
• The second technique is also where data is missing at random basis but here missing values are replaced referring to the other data points available with current dataset.
• Third technique is where data is not missing at random basis and mainly replaced using Mean/Median/Mode values.

‘Disambiguation means to make something clear and understandable.’

We all know the word i.e., Ambiguous which means something which is unclear especially when same words are used for multiple purposes.

In the current era, semi and unstructured data is the norm. Now a days, social media has become one of the major sources for customer profiling. Then audio and videos are another source which can have any kind of data behavior.

‘Data Disambiguation’s biggest use case is to handle Sarcasms.’

As we know, in Sarcasm, dual meaning words are heavily used. For example, there is a common saying i.e., What’s up? This phrase can be used to say greeting to a friend and can also be used to show anger to someone unknown moving around in your restricted area.

The job for Data Disambiguation process is.

• To dig out the right meaning out of confusing statements.
• To understand the right concept of statement.
• Remove anything which can corrupt the behavior of pattern.
• Translate or replace dual meaning word with single meaning words so end user is not confused before taking any decision.

Data Fusion, another term overlapping with many other already in-place terms.

‘Data Fusion is about pulling same nature of data from multiple source systems to co-relate with each other to produce one unified output.’

For example, data for a particular product can come from POS (point of sales), from shops, from online stores etc. At the same time, people can talk about that product on social media, in seminars, webinars etc. Fusion of all those data elements can produce valuable information about that product for sales and marketing teams.

Yes, looking into image you can make out same thing can be achieved in Data Warehouse, Data Mart, Data Lake, Data Lakehouse, Data Vault/ Business Vault etc., and you are not wrong. All above-mentioned Data Store can be used for Data Fusion.

Integration Hub is normally confused with datastores like Data Warehouse, Data Lake, Data Lakehouse etc., which is not correction.

Integration Hub is a single central place from where you can send instructions to internal and external systems to perform some actions. Integration Hub consists of Spokes, each Spoke consists of number of steps for actions.

Generics Steps

• Create actions to be sent out to external systems.
• Run commands on external servers.
• Allow sending messages e.g., Rest, SOAP etc.
• Allow parsers.
• Allow SQL command to databases.

Yes, we do it every day e.g., we create connections with any system e.g., creating a connection with SharePoint then running commands to Delete a file. Consider 100 users want to do the same action, each must go through the same number of steps. Using Integration Hub, Spokes consisting of connections, credentials, and actions to be performed on e.g., SharePoint. Now, all those 100 users can reuse the same Spoke created to delete files in SharePoint.

In this era, social media has surpassed every medium to generate data. Social media is not only about writing messages, it also about sending audio or video messages and the moment audio and video (with voice) comes into picture, NLP comes into action.

‘NLP is about using techniques to understand the way humans communicate.’

Using NLP, code tries to convert audio into text and vice versa. Using NLP capabilities there are solutions which talk to humans, understand what they want and then reply to them. NLP also assists in performing actions e.g., a human can ask ‘I want a Chill Coke Zero’, the NLP will recognize this request and get a Cold Coke Drink for customer. At the same time, if Chill Coke Zero is out of stock, using Machine Learning and AI, it can reply to customer in Voice format that ‘At the moment, we are out of stock, would you like to go for Chill Pepsi Black’.

Steps of NLP

• Understand
• Analyze
• Manipulate
• General Human Language or Text

Initially, I planned to write two separate topics on Data Strategy and Data Management but as both are huge areas and the aim of this book is not to train people but rather to make sure readers understand the topics, so decided to combine.

Data Strategy is about defining many elements to make sure when a data solution is implemented, it covers everything to host data, to manage data, to protect data, to maintain data, to make use of data etc.

‘Data Management is one of the Key Element of Data Strategy.’

As the image depicts itself, Data Strategy is about defining all the features and capabilities before the execution process starts. Data Strategy is about tools, processes, implementation planning, maintaining, planning etc., based on business or organization requirements. Data Strategy covers business strategy, organizational roles, data architecture and data management. Data Strategy goals are to innovate, cater to business requirements and address risk and regulatory challenges.

Hype Cycle of Data Management is confused with Data Management which is not correct as one is about solution and other about enterprise.

Hype Cycle of Data Management is about knowing the pattern of technologies like how past tools and technologies performed, and which are new technologies in trend.

Hype Cycle of Data Management assist organizations to plan its technologies roadmap, it assists to take decisions like which tools and technologies to adopt and which not to adopt.

There are many a times when organizations are reluctant to adopt new technologies and they end up like Kodak and Nokia.

There are many companies which prepare these kinds of statistics and share with CTO(s), CDO(s), CFO(s), CEO(s) etc.

To make it easy, first must understand Traditional Data Stake (TDS). Now, what is TDS lolz 😊. TDS is your On-premises solution which can include any kind of Databases or maybe Hadoop environment as well. Whereas MDS is your tools and technologies on Cloud.

‘MDS vs TDS is more over a comparison between Cloud vs On-premises infrastructures.’

In TDS/ On-premises, if you need new hardware, there is a complete procurement cycle involved. If you need 2 CPU of processing at nighttime and 4 CPU in daytime but you end up paying 4 CPU for 24 hours as you have already bought the hardware 😊.

In MDS, when you are on Cloud, any resource is few clicks away. If you need 2 CPU as night, you only pay when you are using those and same for daytime CPU requirement.