Knowledge Hub

A Data Glossary is a vital step towards mitigating the risk of creating a potential Data Swamp.

Data Glossary vs. Data Dictionary falls under the umbrella of Metadata Management.

• A Data Dictionary is a repository that houses the names, definitions, and attributes of data elements, providing critical contextual information about data.
• A Data Glossary, on the other hand, is a collection of terms that define the key characteristics of your data. These terms are organized in an easy-to-search format to facilitate quick understanding and access.

What Does a Data Glossary Contain?

A Data Glossary includes business terms that provide meaning and context. For instance, a table may have terms like region, sales, quantity, brand, and year. Anyone analysing this table will quickly understand that it relates to sales data, with quantities likely provided brand-wise at a regional level.

A comprehensive Data Glossary can refer to:

A Data Dictionary is the next step towards mitigating the Risk of a potential Data Swamp.

‘Data Dictionary comes under Metadata Management.’

In the IT World, we all know that new products and solutions keep coming year by year, so it's very important to capture what has been implemented. A Data Dictionary is one of the most important items for a system's impact analysis as it gives a clear picture of all the elements in the database.

With the introduction of Big Data where semi and unstructured datasets have come onto surfaces like video, audio, logs, emails, pdf & social media etc., so if one doesn't have a Data Dictionary in place then all future implementations will take 2x-3x more time.

Data Dictionary stores metadata information about databases like.

• List of all the objects and when each was created or updated.
• Details like field name, description, data type, length, indexes, primary & foreign keys etc.
• Relationships between tables, schema(s) and databases etc.
• Number of records in each table

In today’s hyper-competitive digital landscape, the imperative for a robust Enterprise Architecture (EA) framework cannot be overstated. For Top Management, synchronizing technological initiatives with overarching business imperatives is essential to optimizing value creation and operational efficacy.

A holistic EA framework initiates with the essential inquiry of "WHY?", the foundational rationale underpinning each initiative. By accurately revealing the purpose and securing explicit management endorsement, each project is thoroughly aligned with strategic imperatives. Comprehensive guidance is then provided on establishing precise Go-Live timelines, delineating resource requisites, and selecting optimal deployment paradigms, whether on-premises or within a Cloud ecosystem.

A robust EA framework encapsulates four pivotal architectural dimensions. Business Architecture is devoted to harmonizing business exigencies with strategic aspirations, facilitating the development of cogent business strategies, models, and processes that engender substantial value for stakeholders. Data Architecture is predicated on the meticulous organization and management of data, fostering the creation of sophisticated data models, definitions, and flows that yield actionable insights and inform strategic decision-making. Application Architecture addresses application requisites, ensuring that enterprise systems and databases are finely tuned for peak performance and functional integrity, thereby augmenting value for end-users. Lastly, Technical Architecture concentrates on the deployment of requisite physical infrastructure, fortified security protocols, and advanced network solutions to support the technical landscape and guarantee the seamless execution of strategic initiatives.

One must first understand the existing Enterprise Data Architecture Landscape as every new project’s data architecture must follow existing Landscape rather introducing new components with every new project. I have personally seen organizations creating Data Components/ Tools Swamp, as in current era of Big Data there are hundreds of tools offering the same functionalities so one must know whether a tool is already there in existing Landscape or not for a particular requirement. I would consider this as, one the biggest Risk in current era.

Data Architecture is the most important aspect of any data related solution or project. As mentioned in the previous chapter, there are four types of architecture i.e., 1) Business Architecture 2) Data Architecture 3) Application Architecture and 4) Technical Architecture. Most of the organizations confuse or mix-up the last three, which is again a recipe for disaster. Data Architecture should remain standalone from Application or Technology Architectures.

On-Premises and Cloud both compete but at backend both operate the same way.

Since years and years Cloud is trying to make a mark but some or the other reason it couldn’t. Now things have changed, in recent era, Cloud has become arguably the biggest buzzword after the Covid-19 pandemic in 2019/20, as organizations had to enable Cloud services for their employees to support their businesses.

Let’s try to understand the core difference, to use IT solutions there are always two options either to have all the software(s) on On-Premises or on Cloud.

On-Premises: When an organization buys servers and infrastructure, and places those inside their own location or building under their own control, managed by their own internal teams, it is called On-Premises.

Cloud: Now when someone else places servers and infrastructure in their location or in their building and manages it for customer where customer accesses those servers and infrastructure via internet, it is called Cloud.

In IaaS, online compute power and storage is provided to customers in the shape of Virtual Machines (VS). Customers must bring their own software, tools and IT Solutions and then manage the environment, from installing/ upgrading software, HA, DR, compute power, storage standard etc., on its own. Cloud Service provider charges based on compute and storage usage.

Cloud example segregating IaaS:

If you only expect Me (Service Provider) to buy laptops (Infrastructure or Servers) with ONLY having Operating System of your choice and give you access and all rest of the activities like to install your special software(s) on it, install security software(s), enable software encryption, replaces normal hard disk with encryption disk, install all MS Office software(s) on it for daily activities, install database on it, configure Disaster Recovery and High Availability Strategy like taking backup on an external disk and keep moving data on periodic basic on second laptop so just in case if first laptop crashes, work can be continued on second laptop, periodic upgrading all the software(s) on both laptops etc. etc. etc. all will be take care by yourself is IaaS.

We have discussed differences between On-Premises and Cloud then IaaS where only Hardware is rented by Cloud providers. Next stage of this paradigm change is to rent Containers at a small scale, to save cost and for better management of resources. CaaS (Container as A Service) is one level lower than IaaS to rent resources on Cloud.

As the name says itself, CaaS (Container as A Service) is about deploying code in a container and deploying that container in Cloud. This saves us to manage hardware on our own in Cloud i.e., IaaS, so we can deploy our containers and underneath it, the hardware will be managed by Cloud providers. Most of the Cloud providers are offering CaaS.

You are using Agile Methodology then you need CI/ CD (explained separately). In Agile, most of the time we have multiple teams working in silos on different modules on the same project and can be on the same deliverable. CaaS can be very helpful in this methodology where different teams are able to develop and deploy their code set any time using CI/ CD pipelines.

As we move from IaaS to CaaS to PaaS, it becomes confusing. In IaaS and CaaS, we established that Infrastructure/ Servers and Storage is provided by Cloud providers and rest of all the activities are managed by customer team, itself. Now, let’s try to understand PaaS where all software(s) are made available for customers by the Cloud provider. Software(s) are already installed and ready to use. Customers don’t have to manage any compute or storage; everything is managed automatically by Cloud provider.

If you expect Me (Service Provider) to buy laptops (Infrastructure or Servers) with all the possible software(s) pre-installed and made those available to you via Internet, ready to be used in few clicks. Along with software(s) all rest of the activities like to install security software(s), enable software(s) encryption, replaces normal hard disk with encryption disk, install all MS Office software(s) on it for daily activities, install database on it, configure Disaster Recovery and High Availability Strategy like taking backup on an external disk and keep moving data on periodic basic on second laptop so just in case if first laptop crashes, work can be continued on second laptop, periodic upgrading all the software(s) on both laptops etc., all will be taken care by Me (Cloud Provider) is PaaS.

We have discussed IaaS and CaaS where only Infrastructure and Container level resources are rented on Cloud. Next stage of this paradigm change is to deploy Functions ONLY and pay for resources required to run those only. FaaS (Function as A Service) is one level lower than CaaS to rent resources on Cloud.

One of the biggest challenges to date is, whenever there is a change in the requirement, the whole code-set must redeploy. When code changes, it was always challenging to redeploy ONLY the changed code as there are always dependencies. Breaking the bigger code set into smaller pieces of code sets i.e., functions, can assist in mitigating this challenge wherever possible. Agree, in Data Pipelines it might be difficult but worth having this option to be utilized whenever possible.

As the name says itself, FaaS (Function as A Service) is about deploying ONLY Functions in Cloud. This saves us to manage Infrastructure on our own in Cloud, now we can deploy our Functions ONLY and underneath it the infrastructure will be managed by Cloud providers. Most of the Cloud providers are offering FaaS.

As we move from IaaS and PaaS to SaaS, it becomes very confusing segregating PaaS and SaaS, but please hold on to your horses’ 😊 we are going to nail it down in this topic.

To understand this example, pre-requisite topics are 1) ‘On-Promises vs Cloud’ and 2) IaaS and 3) PaaS.

In IaaS, we established that only Infrastructure/ Servers and Storage is provided by Cloud providers then in PaaS we establish along with Infrastructure/ Servers and Storage, Software(s) BUT EMPTY Software(s) are also made available along with rest of all the activities to be managed by the Cloud providers EXCEPT NO IMPLEMENTED SOLUTION.

In SaaS, there are pre-built or OOTB Solutions available in the Cloud. Please note, it’s not mandatory that all solutions are owned by Cloud providers, most of the solutions are created and maintained by the third parties. Third Parties use compute and storage from Cloud providers and build their own solutions and sell to customers. Now, customers just need to log-in into the Solution in Cloud as a Service that is why it’s called Software as A Service. A solution is owned and its management like installation/ upgradation, compute, storage, HA, DR etc., all are taken care of by the Solution Owner, again which can be by Cloud Providers or any Third Parties. Customers just need to subscribe to the SaaS service and start using the Solution.