Knowledge Hub

Data Discrepancies are confused with Data Errors which is not true. Data Errors are searched, identified, and fixed based on Data Discrepancies.

‘Dictionary: An illogical or surprising lack of compatibility or similarity between two or more facts are called discrepancies.’

Data discrepancies can be like mismatch of data from source verses data processed.

Some time, when data analysts run manual calculation, those are not matching with the numbers coming from automated reports which means there are data reconciliation issues. This is Data Discrepancy.

For example, two results do not match based on business logic. Refer to the image, total of 9,093 + 6,104 is not equal to All Users i.e., 14,018. This means, a total of 14,018 came from various sources and 9,093 & 6,104 came from different sources. This is Data Discrepancy.

As mentioned above, once data discrepancies are found, the next step is to start identifying the data errors, code errors, business rule errors etc.

Data Imputation is majorly used in Data Science domain.

‘Imputation means, fill missing data.’

Data Imputation normally comes into play so business rules can be imposed on all data rather on subsets of data. Before Machine Learning Model runs, data imputation is done by either missing data is filled up or the whole row is removed.

Missing value can create many kinds of issues e.g., during categorizing, classifying, clustering etc., if data is missing then results will not be 100%.

Generally, in numeric datasets, missing values are replaced with MEAN values. Having said that, there many ways to deal with missing data.

Few Imputations Techniques

• The first technique is where data is missing at random basis but before using it, it is completely removed from the dataset.
• The second technique is also where data is missing at random basis but here missing values are replaced referring to the other data points available with current dataset.
• Third technique is where data is not missing at random basis and mainly replaced using Mean/Median/Mode values.

‘Disambiguation means to make something clear and understandable.’

We all know the word i.e., Ambiguous which means something which is unclear especially when same words are used for multiple purposes.

In the current era, semi and unstructured data is the norm. Now a days, social media has become one of the major sources for customer profiling. Then audio and videos are another source which can have any kind of data behavior.

‘Data Disambiguation’s biggest use case is to handle Sarcasms.’

As we know, in Sarcasm, dual meaning words are heavily used. For example, there is a common saying i.e., What’s up? This phrase can be used to say greeting to a friend and can also be used to show anger to someone unknown moving around in your restricted area.

The job for Data Disambiguation process is.

• To dig out the right meaning out of confusing statements.
• To understand the right concept of statement.
• Remove anything which can corrupt the behavior of pattern.
• Translate or replace dual meaning word with single meaning words so end user is not confused before taking any decision.

These are one of the most confusing topics. We all mix-up data quality fixes and standardization of data for better processing with transforming data for better processing and serving.

Let’s decode it…..

We all know, in Data Warehouse or when we pull data from an OLTP system, we first store it in a LANDING ZONE somewhere which normally is the true copy of source systems. Then we move it to STAGING ZONE before we transform it into a Unified Data Warehouse (Bill Inmon way) or in a Dimensional Model (Ralph Kimble way).

Tables in LANDING and STANGING ZONES, both keep the true DATA MODEL from source systems but not the CONDITION of DATA.

WHAT DOES ABOVE STATEMENT MEANS, not the CONDITION of DATA?

Data Standardization

Does standardizing the condition of data not equal to transformation? NOPS, ITS NOT. Data Standardization can have unlimited scenarios, few are shared below e.g.,

• Standardization of Dates:
  • Converting different dates formats from different source systems e.g., DD/MM/YY, MM/DD/YY, YY/MM/DD, YY/DD/MM etc., into one format e.g., DD/MM/YYYY.
  • Convert empty date rows into e.g., 01/01/9999.
• Convert special characters to NA.
• Convert empty fields to NA.
• Profile the fields, if data type is string or varchar but data is only number than convert it into digit or number as running queries on digit or numbers are way faster.
• If the field contains codes only with only character, then make it CHAR and if it only contains numbers then make it DIGIT.
• If the full name of a person is in one field, convert it in First, Middle and Last Name.
• Fix the spelling mistakes.
• Fix the up and lower cases e.g., if a country is Australia and in data its coming as Aus, Aust, australia, Aussie, aus etc., than convert it in one format e.g., Australia.

In the context of the data world, "BASEL" likely refers to the Basel Committee on Banking Supervision. The Basel Committee is not directly related to data management but plays a significant role in shaping international banking regulations, including requirements related to risk management and data reporting.

Basel Committee on Banking Supervision (BCBS) is an international committee of banking supervisory authorities. It was established by the Bank for International Settlements (BIS) to strengthen the regulation, supervision, and practices of banks worldwide. The committee is known for developing and maintaining global regulatory frameworks, commonly known as the Basel Accords, which set standards for bank capital adequacy, risk management, and prudential supervision.

Basel Accords/ Standards:

• Basel I: Introduced in 1988, Basel I established minimum capital requirements based on credit risk.
• Basel II: Developed in 2004, Basel II expanded the framework to address credit, operational, and market risks. It introduced more risk-sensitive capital requirements.
• Basel III: Implemented in response to the 2008 financial crisis, Basel III further strengthened capital requirements, introduced new liquidity standards, and enhanced risk management practices.

Testing and Quality Assurance (QA) are commonly mixed up, but let's break down the differences.

Testing:

Testing is about discovering problems or bugs in processes or code, either during or after implementation. Various stages of testing happen during an IT project, such as Unit Testing (checking individual parts), System Integration Testing (verifying components work together), and User Acceptance Testing (ensuring the solution meets user requirements).

QA (Quality Assurance):

QA, on the other hand, is all about making sure that the standards and output of processes or solutions meet customer satisfaction. Even if testing doesn't find bugs, QA checks if the output aligns with the expected quality. For example, if we create a table with a million rows, testing may show all rows are there, but QA might reveal differences in country name spellings or data formats.

For example, imagine you're building a database with country names. Testing might confirm that all million rows are there, but QA would catch if the spellings were inconsistent or if data formats are not uniform.

Housekeeping refers to the routine maintenance, organization, and management tasks that are necessary to ensure data quality, integrity, and efficiency. It involves activities such as cleaning, organizing, and optimizing data to keep databases, data warehouses, and other storage systems in good order.

With an optimistic outlook, effective housekeeping measures have the potential to save approximately 30% of operational costs.

Here are some examples of housekeeping tasks in the Data World:

• Data Cleaning: Identifying and rectifying errors, inconsistencies, or missing values in datasets. For example, if you have a dataset with student information, data cleaning may involve fixing typos in names, standardizing date formats, and filling in missing grades.
• Data Deletion: Removing obsolete or unnecessary data to free up storage space and reduce clutter. For example, deleting records of students who have graduated or left the school to keep the dataset relevant and manageable.
• Backup and Recovery: Regularly creating copies of data to prevent loss and establishing procedures for data recovery in case of accidental deletion or system failure. For example, saving multiple copies of your project work on different devices to avoid losing progress if a file becomes corrupted.
• Data Archiving: Moving older or less frequently used data to long-term storage to optimize performance in the primary data storage system. For example, archiving past school year records to a separate folder to keep the current year's data easily accessible.
• Indexing: Creating indexes on databases to accelerate data retrieval and query performance. For example, indexing a list of books by title or author's name for quicker reference when searching for specific information.
• Data Encryption: Applying encryption to sensitive data to protect it from unauthorized access. For example, using encryption tools to secure personal information in a project related to online privacy.
• Metadata Management: Organizing and documenting metadata (data about data) to enhance understanding and usability. For example, creating a spreadsheet that documents the meaning and format of each column in a dataset.
• Data Purging: Removing data that is no longer needed or relevant, especially in compliance with data retention policies. For example, deleting survey responses from previous years to keep the dataset focused on recent data.
• Performance Optimization: Fine-tuning databases and queries to improve overall system performance. For example, adjusting settings in a gaming app to improve its performance on a smartphone.

At its core, functional testing involves validating all the use cases and requirements outlined in the Functional Requirements Document (FRD) or other business requirement specifications to ensure the product operates as intended. In contrast, non-functional testing focuses on evaluating the product's quality attributes, such as performance, compliance with governance standards, security measures, and overall system reliability, ensuring it meets predefined benchmarks beyond functional correctness.

Testing is a critical phase in the software development lifecycle, ensuring that a product meets its requirements and functions as intended. It is broadly divided into two categories: functional testing and non-functional testing. Understanding these categories is essential for developing robust, reliable, and user-friendly software systems. This chapter explores these two testing types, their significance, methodologies, examples, and key differences.