Data Observability

Share this topic

Data Observability

From the book I Am Datapedia!, published here free by the author.

Part of I Am Datapedia!

Diagram, venn diagram

Description automatically generatedData Observability is one of the most underrated terms, as its name says, it's about observing and providing analysis and analytics for decision making.

‘Data Observability comes under Metadata Management.’

Yes, Data Observability aims to focus on observing every data pipeline, data transformation, DDL, DML, data governance standards, security & privacy policies etc. across the organization. It creates a holistic view of everything, it assists to find out where the problem is and debugging steps assist to make future decisions, so the same problem don't occur in future.

Pillars of Data Observability

  1. Freshness
  2. Lineage
  3. Schema
  4. Volume
  5. Distribution

Now the question is, observe but how? Do we or can we give access to one framework on all the frameworks in Enterprise? Of course, the Security, Risk, Compliance and Audit departments won't let it happen. But NOT-IF the access is requested only on tools for Metadata Management. Yes, Data Observability mainly access Enterprise Metadata Data.

It observes.

  • Metadata of all the tables in all the data stores
  • All the up and downstream datastores and applications
  • On request, it observes sample data for debugging purposes.
  • Dashboards, Reports, Code Sets, KPI(s) etc.
  • All the ETL/ ELT tools and shell scripts
  • All the logs
  • Etc.

Next question can be, even if access is granted to all Metadata Repositories across Enterprise, is it possible to implement it when all the frameworks are sitting in silos, in many clusters, in many networking segments, in different network zones, in multiple environments, each of which is supported by firewalls etc. So, in the IT World where there is no centralized control, Data Observability will be a challenge.

‘Ideally, Enterprise Data Observability is possible with Data & AI Cognitive (DAC) Architecture in-place.

Express your reaction

Your Take on This Insight?

Share how this content resonates with you.

Comment
Useful0
Insightful0
Actionable0
Thought-provoking0
Inspiring0

0 reactionsSign in to react or comment

Comments (0)

No comments yet.

    Free account

    You can read every topic without one. A free account lets you react to this topic, comment under it and search the Knowledge Hub. Reacting and commenting need a confirmed e-mail address.