Knowledge Hub

Visionaries and business users thought that because they understand their business, know their problems, understand their use cases, and now have Vibe Coding and AI-assisted development at their fingertips, they could build things themselves.

So they began asking:

"Why do we need Data people? We understand the business. We know what we want to build. And AI can help us build it."

And to be fair, the early results look promising.

POCs went well.
Pilots went well.
Applications were built. Dashboards were created. Workflows were automated.

Everyone is excited.

But now comes the interesting part.

• Now it's time to bring in the actual enterprise data.

• Now it's time for proper user acceptance testing.

• Now it's time to fix data quality issues that nobody noticed during the POC.

• Now it's time to optimise the code because compute consumption is increasing, cloud costs are approaching budget limits, and the expected ROI is taking a hit.

• Now it's time to optimise storage, remove unnecessary data duplication, and manage growing volumes.

• Now it's time to integrate with existing enterprise systems, deal with incompatible data structures, and resolve dependencies.

• Now it's time to migrate from legacy technologies to newer platforms without disrupting business operations.

• Now regulatory bodies are asking questions. We need to produce compliance reports, demonstrate data lineage, maintain audit trails, and prove where the data came from and how it was transformed.

• Now business leaders want transparent and trusted KPIs. We need to reconcile different definitions, validate calculations, and ensure everyone is looking at the same numbers.

I have seen organisations invest years building data platforms, introducing new technologies, and expanding their enterprise architecture. Over time, these investments become part of the technology landscape.

Then something interesting happens.

A new platform is introduced, but the old one stays. A new data warehouse is built, but the previous warehouse continues running. A new reporting solution is launched, but business users still depend on the old reports. Another integration layer is added because replacing the existing one is considered too difficult.

Eventually, the organisation is operating multiple generations of technology, often solving similar problems in slightly different ways.

And nobody really wants to switch anything off.

There are understandable reasons. Dependencies are complicated, business users are comfortable with existing systems, migration carries risk, and nobody wants to be responsible for an outage.

Epistemology, a term rooted in the Greek word episteme, meaning “knowledge,” and logos, meaning “discourse” or “communication,” explores the concept of “justified knowledge.” It addresses questions like “What do we know?”, “How can we claim to know something?”, “What validates our beliefs?”, and “How do we ascertain the certainty of our knowledge?”

In the realm of data, these questions take on a new dimension, especially in the Big Data era, where we store vast amounts of raw data, structured, semi-structured, and unstructured. Data science teams apply machine learning or deep learning models to extract insights, turning raw data into knowledge. However, to ensure that these insights align with their origins, a “justification of knowledge” is essential, ensuring that what we extract truly reflects what was expected. This process emphasizes evidence-based validation, where qualitative research relies on the quality of design and data collected.

Scope Creep is a known term among Project Management but is worth discussing it among data folk as scope creep can also impact data architecture and data modelling.

‘Scope Creep are those new requirements which are never formally added in project scope or if added, are without estimating impact of project deadline.’

Scope Creep means adding new requirements without analyzing impact on time and cost. I personally follow the thumb rule of TCS.

‘TCS: Time, Cost and Scope.’

Whenever 1 of the above mentioned 3 pillars of project management is impacted, the remaining 2 WILL also be impacted. Scope Creep is referred as e.g., a new requirement is added to the project scope, but the Project Manager didn’t estimate how much more effort is required and will there be any additional cost involving it. At the end of the day, it may impact Project Go-Live date which no one wants.

Reasons of Scope Creeps

• Deficiency of a Vibrant and Thorough Scope
• Absence of a Stakeholders Concurrence 
• Weak Project Sponsor Decision Management 
• Contradictory Business Users Views
• Less participation of all stakeholders during Project Lifecycle
• Bad Management of Risk and Issues Proactively
• Crossing Prioritizing Requirements
• Lack of Change Management Process
• Bad Efforts Estimation
• Bad Evaluation of new Requirements
• Missing Stakeholders
• Non-identification of Out-of-Scope earlier in Project Lifecycle
• Vague Acceptance Criteria(s)
• Poor Assumptions Management
• Poor Constraints Management
• No Clear Project Deliverables

Hype Cycle of Data Management is confused with Data Management which is not correct as one is about solution and other about enterprise.

Hype Cycle of Data Management is about knowing the pattern of technologies like how past tools and technologies performed, and which are new technologies in trend.

Hype Cycle of Data Management assist organizations to plan its technologies roadmap, it assists to take decisions like which tools and technologies to adopt and which not to adopt.

There are many a times when organizations are reluctant to adopt new technologies and they end up like Kodak and Nokia.

There are many companies which prepare these kinds of statistics and share with CTO(s), CDO(s), CFO(s), CEO(s) etc.

Foresight and Futurology are used interchangeably. And yes you are right this new term is not something we have never encountered before.

Let’s decode it…..

Data: Refers to information, facts, or statistics collected for analysis, reference, or computation.

Futurology: It's the study or soothsaying of possible futures. It involves examining current trends and making prognostications about unborn developments.

Combining these," Data Futurology" suggests concentrating on the future of data, in terms of how data is collected, reused, anatomized, and its impact on colorful aspects of society or technology.

Let go little deeper about conception of" Data Futurology," which involves the study and soothsaying of possible futures related to data.

Let's break it down:

• Study or Forecasting of Possible Futures
• This refers to the practice of examining current trends and making predictions about future developments. It involves analyzing data and information to anticipate how various aspects of a particular field might evolve over time.
• Data Futurology
• This term specifically focuses on the future of data. It encompasses the entire data lifecycle, including how data is collected, processed, and analyzed. Additionally, it explores the potential impact of data on different aspects of society and technology.

POC stands for Proof of Concept. POC normally follows Sales Context of the product.

In POC, output is most important, not technology. In POC, the implementers make sure output is fulfilling customer requirements and at the same time convince business that this is the solution to go with.

The Goal of POC is to make sure whether Idea is Feasible or not.

‘POC is built and run for Business Users.’

Protype is the next stage of POC. In POC, one has test that yes Idea is Feasible, and it will benefit business agenda.

Now, in Prototyping, tools and technologies comes into picture where one starts converting Ideas in Reality, where it will show how the end-result will look like. It will show, how product will look like. It will show the real end solution.

Prototype is not built considering that business users will jump in and start using it. Prototypes are normally for technical folks to evaluate the end results with dumpy but relevant data.

’Prototype is built and run for Technical Users.’

New name of Pilot is MVP 😊. MVP is the next stage of Prototyping where Tools and Technologies are tested, and end solution is tested by technical folks. Now in MVP, subset of actual requirements is considered, subset of real data is used, subset of hardware and infrastructure is used, subset of software is used, and limited number of features are implemented.

Once MVP is ready, it’s released to actual users to start using it. This gives business users a taste of the actual solution. The biggest advantage of this approach is, with small implementation, with small investment, business can taste the blood and take a decision to go to the next stage or drop the solution.

Now a days, many organizations divide complete solution into multiple MVP(s) i.e., by implementing solutions in chunks, this is also supported by Agile approach where product gets ready in shorter timeframes and then keeps maturing with time. In MVP, just like actual implementation, three environments are spin i.e., DEV, UAT and PROD, along with all the steps like DevOps, CI/CD, Metadata Management, Data Quality, Data Governance, Data Lineage, Master and Reference Data Management, and many other steps which are required to go to Production.

The terms "IT" (Information Technology) and "Digital" are related but refer to different concepts within the broader technological landscape.

Information Technology (IT) encompasses the use, development, and management of computer systems, software, networks, and information. It is primarily concerned with the technology used to store, retrieve, transmit, and manipulate data. For example, IT includes hardware (computers, servers, storage), software (applications, operating systems), networking (local area networks, wide area networks), and the management of databases and information systems. IT is essential for the efficient storage, processing, and communication of information within organizations.

Digital is a broader term that extends beyond just technology and includes the transformation of processes, businesses, and society through the use of digital technologies. It is concerned with the integration of digital technologies into various aspects of life and business to improve efficiency, create new opportunities, and enhance experiences. For example, Digital transformation involves not only IT but also areas such as data analytics, artificial intelligence, internet of things (IoT), cloud computing, and other emerging technologies. Digital initiatives aim to leverage technology to bring about fundamental changes in how organizations operate, interact with customers, and deliver value.