Metadata Management

Metadata Strategy should take place along with project or product strategies, means the earlier the better.

We stated in the Master Data Management term, can Metadata & Master Data reduce your BI footprint by 80%? Yes, it can. For example, if you produce a dashboard/report and describe its Metadata in such a manner that it articulates the entire story around the data points and the KPIs in it then the next BI report will only be generated if the earlier one doesn’t aid the purpose.

In a generic sense we say, Metadata is moreover Data-About-Data, but Metadata is beyond Data-About-Data. It can tell you about business requirements, it can tell you about business goals, it can tell you about Business KPIs etc.

We all have read books either in our student lives or professionally. The Index of the book is Metadata which indicates which page contains which topic. The Acknowledgment page of the book is Metadata as well which tells the overall background, reason, agenda etc., about the book. Metadata can host roles, grants, business processes, terms and conditions, data points and KPI(s) on dashboards and reports, details about databases, tables, columns and rows, the definition of the tables, description of the columns etc. If your Metadata is not in place, you can’t figure out Data Lineage. If your Data Lineage is not placed, your Data will become a Data Swamp, so on…. And again, you are expendable☺. Now is the era of Big Data and Data Lakes, social media, and semi and unstructured data. If one builds a Data Lake without Metadata Management in place, good luck to the organization.

I believe organizations should also start planning to build BI solutions just on Metadata for BAU support.

Types of Metadata

• Business Metadata: It includes metadata about the solution implementation e.g., like business definition, descriptions, data model, data quality rules, data standards etc.
• Technical Metadata: It includes metadata about what has been implemented in solution e.g., database/ table/ column names, indexes, access permissions, ETL jobs details, data lineage documentations etc.
• Operational Metadata: It includes metadata about the execution result of implemented solution e.g., execution date, error alert, job duration etc.

Sources of Metadata

There can be N number of sources to gather Metadata e.g., Applications, Data/ Business Glossary, Data Dictionaries, Systems Catalogs, Database Management Catalogs, Reporting Tools (BI), Tools for Configurations, ETL or Integration Tools, Data Mapping Excels, Data Quality Tools, Event Management Tools, Data Modelling Tools, Reference Data, Service Registries, Source lists, Interface details, Code sets, Spatial schemas, Graphical Datasets, etc.

Metadata Architecture Components

• Metamodel: Its concept is same as traditional data modelling where first high level conceptual metamodel can be created to show connections between systems, followed by low level metamodel to show attributes, data elements, processes etc.
• Metadata Portal: It is a layer exposed for user to search metadata.
• Enterprise Metadata Repository: This is a centralized repository which can host metadata from all the source systems for users to search for required metadata. Please note, not all the metadata architectures Enterprise Metadata Repository is loaded.
• Source Metadata Repositories: These are the individual repositories hosting metadata at source side. Sources can be many as mentioned in previous section.

Type of Metadata Architectures

• Centralized Metadata Architecture: In this design, there is a centralized enterprise metadata repository which is loaded from all the available source system metadata repositories. Metadata portal connects with this enterprise metadata repository for all kinds of metadata. Cross systems metadata search is possible in this architecture.
• Distributed Metadata Architecture: In this design, there is no enterprise metadata repository, so metadata portal is directly connected with all source metadata repositories. In this architecture, cross systems metadata search is not available.
• Hybrid Metadata Architecture: This design consists of both enterprise metadata repository and distributed metadata architecture. This architecture is more beneficial where source metadata is rapidly changing and experience high growth in metadata and metadata source systems.
• Bi-Directional Metadata Architecture: This is the most complex architecture where changes are Bidirectional, means metadata is pulled from source systems and when updated in enterprise metadata repository, it pushed back to source systems. This way both repositories are up to date all the time.