Data Profiling

Data Profiling is used by Data Modelers, Data Engineers, Data Governance and Data Quality team as their daily routine.

Data Profile is done using many methods to summarize data for analysis. The aim is to find out the data patterns.

For example, if you want to find out, is there a person in your customer list whose age is above 90 years? Now, one way is to go through each line of data and find out who is aged more than 90 years, or the other way is to run a DINTINCT command on age column which will give you all-unique ages from age column. Consider, if there are customers from age 21 to 100 in your database then unique values should not be more than 80.

Data Profiling is also heavily used by Data Quality teams to find out whether data is following standards defined by Data Quality teams. For example, there is Data Quality Rule i.e., all date columns in the system must follow date format as DD/MM/YYYY. Now, as data is flowing-in on daily basis and sometimes in real-time manner, Data Quality teams have automated Data Profiling functions running to run DISTINCT commands to see if there is any other date format coming in the data. Based on the above date format example, there should always be one unique row from date fields.

Data Profiling Key High-Level Steps

• Collecting Data
• Cleansing Data
• Run Profiling Tasks

Data Profiling Tasks Involves

• Gather details like min, max and sum of the data.
• Gather details like datatype, lengths of columns.
• Find out regular patterns.
• Find out keywords, descriptions, or categories by using distinct and group by commands.
• Run data quality assessment.
• Records metadata, data dictionary and data catalog
• Classifying candidate columns for primary and foreign keys, for uniqueness, for composite keys etc.