Data Observability

Data Observability is one of the most underrated terms, as its name says, it's about observing and providing analysis and analytics for decision making.

‘Data Observability comes under Metadata Management.’

Yes, Data Observability aims to focus on observing every data pipeline, data transformation, DDL, DML, data governance standards, security & privacy policies etc. across the organization. It creates a holistic view of everything, it assists to find out where the problem is and debugging steps assist to make future decisions, so the same problem don't occur in future.

Pillars of Data Observability

1. Freshness
2. Lineage
3. Schema
4. Volume
5. Distribution

Now the question is, observe but how? Do we or can we give access to one framework on all the frameworks in Enterprise? Of course, the Security, Risk, Compliance and Audit departments won't let it happen. But NOT-IF the access is requested only on tools for Metadata Management. Yes, Data Observability mainly access Enterprise Metadata Data.

It observes.

• Metadata of all the tables in all the data stores
• All the up and downstream datastores and applications
• On request, it observes sample data for debugging purposes.
• Dashboards, Reports, Code Sets, KPI(s) etc.
• All the ETL/ ELT tools and shell scripts
• All the logs
• Etc.

Next question can be, even if access is granted to all Metadata Repositories across Enterprise, is it possible to implement it when all the frameworks are sitting in silos, in many clusters, in many networking segments, in different network zones, in multiple environments, each of which is supported by firewalls etc. So, in the IT World where there is no centralized control, Data Observability will be a challenge.

‘Ideally, Enterprise Data Observability is possible with Data & AI Cognitive (DAC) Architecture in-place.