Knowledge Hub

Scope Creep is a known term among Project Management but is worth discussing it among data folk as scope creep can also impact data architecture and data modelling.

‘Scope Creep are those new requirements which are never formally added in project scope or if added, are without estimating impact of project deadline.’

Scope Creep means adding new requirements without analyzing impact on time and cost. I personally follow the thumb rule of TCS.

‘TCS: Time, Cost and Scope.’

Whenever 1 of the above mentioned 3 pillars of project management is impacted, the remaining 2 WILL also be impacted. Scope Creep is referred as e.g., a new requirement is added to the project scope, but the Project Manager didn’t estimate how much more effort is required and will there be any additional cost involving it. At the end of the day, it may impact Project Go-Live date which no one wants.

Reasons of Scope Creeps

• Deficiency of a Vibrant and Thorough Scope
• Absence of a Stakeholders Concurrence 
• Weak Project Sponsor Decision Management 
• Contradictory Business Users Views
• Less participation of all stakeholders during Project Lifecycle
• Bad Management of Risk and Issues Proactively
• Crossing Prioritizing Requirements
• Lack of Change Management Process
• Bad Efforts Estimation
• Bad Evaluation of new Requirements
• Missing Stakeholders
• Non-identification of Out-of-Scope earlier in Project Lifecycle
• Vague Acceptance Criteria(s)
• Poor Assumptions Management
• Poor Constraints Management
• No Clear Project Deliverables

Data Discrepancies are confused with Data Errors which is not true. Data Errors are searched, identified, and fixed based on Data Discrepancies.

‘Dictionary: An illogical or surprising lack of compatibility or similarity between two or more facts are called discrepancies.’

Data discrepancies can be like mismatch of data from source verses data processed.

Some time, when data analysts run manual calculation, those are not matching with the numbers coming from automated reports which means there are data reconciliation issues. This is Data Discrepancy.

For example, two results do not match based on business logic. Refer to the image, total of 9,093 + 6,104 is not equal to All Users i.e., 14,018. This means, a total of 14,018 came from various sources and 9,093 & 6,104 came from different sources. This is Data Discrepancy.

As mentioned above, once data discrepancies are found, the next step is to start identifying the data errors, code errors, business rule errors etc.

Data Skew Issue is one of the main hidden culprits for query performance 😊. It is normally referred to as Skewed data distribution which is used in Data Science domain.

Have you ever noticed, your query is correct, your joins are correct, your all indexes are enabled, you have high data quality, you have data standardization in place but still your query performance is not as expected? You noticed when you loaded new datasets, queries were flying but as time passed, the performance has gone down. You have added more CPU, more RAM, more nodes but NO performance is not like newly added data. It’s because of the Data Skew Issue.

Data Skew Issue is about when data is not distribution evenly. Uneven distribution of data at cluster level causes this issue. It heavily degrades the performance of the queries especially those with joins.

We all have faced a situation where query run exceptionally good till 99% and then hangs for 3-4 hours right lolz yeah, I can understand your pain 😊. Let me tell you the reason, it’s because of Data Skew Issue.

Coupling is a parameter dependency approach used cross software or cross systems.

This approach is used when there are more than one software or system is in the picture to deliver the result.

Data Coupling is about when data elements are used, to kick-off the next step in a different system. For example, in data pipeline, we are pulling data from a hospital management system e.g., when a baby is born let’s have a data dependency like, the moment the baby is registered in system, pull its data. So, it’s a cross-system data dependency or we called Data Coupling.

The disadvantage of Data Coupling approach is, if something goes wrong in the first system, dependent systems cannot move.

Let me share another old example. Old data pipeline folks will remember this one i.e., when we used to load files from another system, we had to wait till the time whole file movement is completed. The challenge was how to make sure when to start pulling like only once file is completely copied.

Data Imputation is majorly used in Data Science domain.

‘Imputation means, fill missing data.’

Data Imputation normally comes into play so business rules can be imposed on all data rather on subsets of data. Before Machine Learning Model runs, data imputation is done by either missing data is filled up or the whole row is removed.

Missing value can create many kinds of issues e.g., during categorizing, classifying, clustering etc., if data is missing then results will not be 100%.

Generally, in numeric datasets, missing values are replaced with MEAN values. Having said that, there many ways to deal with missing data.

Few Imputations Techniques

• The first technique is where data is missing at random basis but before using it, it is completely removed from the dataset.
• The second technique is also where data is missing at random basis but here missing values are replaced referring to the other data points available with current dataset.
• Third technique is where data is not missing at random basis and mainly replaced using Mean/Median/Mode values.

‘Disambiguation means to make something clear and understandable.’

We all know the word i.e., Ambiguous which means something which is unclear especially when same words are used for multiple purposes.

In the current era, semi and unstructured data is the norm. Now a days, social media has become one of the major sources for customer profiling. Then audio and videos are another source which can have any kind of data behavior.

‘Data Disambiguation’s biggest use case is to handle Sarcasms.’

As we know, in Sarcasm, dual meaning words are heavily used. For example, there is a common saying i.e., What’s up? This phrase can be used to say greeting to a friend and can also be used to show anger to someone unknown moving around in your restricted area.

The job for Data Disambiguation process is.

• To dig out the right meaning out of confusing statements.
• To understand the right concept of statement.
• Remove anything which can corrupt the behavior of pattern.
• Translate or replace dual meaning word with single meaning words so end user is not confused before taking any decision.

Data Fusion, another term overlapping with many other already in-place terms.

‘Data Fusion is about pulling same nature of data from multiple source systems to co-relate with each other to produce one unified output.’

For example, data for a particular product can come from POS (point of sales), from shops, from online stores etc. At the same time, people can talk about that product on social media, in seminars, webinars etc. Fusion of all those data elements can produce valuable information about that product for sales and marketing teams.

Yes, looking into image you can make out same thing can be achieved in Data Warehouse, Data Mart, Data Lake, Data Lakehouse, Data Vault/ Business Vault etc., and you are not wrong. All above-mentioned Data Store can be used for Data Fusion.

Integration Hub is normally confused with datastores like Data Warehouse, Data Lake, Data Lakehouse etc., which is not correction.

Integration Hub is a single central place from where you can send instructions to internal and external systems to perform some actions. Integration Hub consists of Spokes, each Spoke consists of number of steps for actions.

Generics Steps

• Create actions to be sent out to external systems.
• Run commands on external servers.
• Allow sending messages e.g., Rest, SOAP etc.
• Allow parsers.
• Allow SQL command to databases.

Yes, we do it every day e.g., we create connections with any system e.g., creating a connection with SharePoint then running commands to Delete a file. Consider 100 users want to do the same action, each must go through the same number of steps. Using Integration Hub, Spokes consisting of connections, credentials, and actions to be performed on e.g., SharePoint. Now, all those 100 users can reuse the same Spoke created to delete files in SharePoint.

In this era, social media has surpassed every medium to generate data. Social media is not only about writing messages, it also about sending audio or video messages and the moment audio and video (with voice) comes into picture, NLP comes into action.

‘NLP is about using techniques to understand the way humans communicate.’

Using NLP, code tries to convert audio into text and vice versa. Using NLP capabilities there are solutions which talk to humans, understand what they want and then reply to them. NLP also assists in performing actions e.g., a human can ask ‘I want a Chill Coke Zero’, the NLP will recognize this request and get a Cold Coke Drink for customer. At the same time, if Chill Coke Zero is out of stock, using Machine Learning and AI, it can reply to customer in Voice format that ‘At the moment, we are out of stock, would you like to go for Chill Pepsi Black’.

Steps of NLP

• Understand
• Analyze
• Manipulate
• General Human Language or Text

Initially, I planned to write two separate topics on Data Strategy and Data Management but as both are huge areas and the aim of this book is not to train people but rather to make sure readers understand the topics, so decided to combine.

Data Strategy is about defining many elements to make sure when a data solution is implemented, it covers everything to host data, to manage data, to protect data, to maintain data, to make use of data etc.

‘Data Management is one of the Key Element of Data Strategy.’

As the image depicts itself, Data Strategy is about defining all the features and capabilities before the execution process starts. Data Strategy is about tools, processes, implementation planning, maintaining, planning etc., based on business or organization requirements. Data Strategy covers business strategy, organizational roles, data architecture and data management. Data Strategy goals are to innovate, cater to business requirements and address risk and regulatory challenges.