Dark Data and Dormant Data

Dormant Data and Dark Data are mostly confused with each other.

Dark Data

‘Data that is not utilized is Dark Data.’

Dark Data is data that is not in use due to lack of technology and is not a Data Asset. Dormant Data is the data, which is useable, but decision-makers don't use it, or it’s not required for analysis but that doesn't mean you can delete it or purge it, so we keep it in Tape drives, in Hadoop and nowadays Object Storage is also good choice.

Before the introduction of Big Data, the % age of Dark Data was around 90% as it included semi and unstructured datasets and sometimes structured as well. For example, JSON or XML files, Video or Audio files, Logs, Metadata etc., all these datasets were there but technologies were either not there or the affair to analyze those were very expensive. The only reason companies kept this kind of data was for Compliance or Audit purposes. Since the Big Data ecosystem has come onto the surface and hardware has become dead cheap so hosting and processing these datasets has become possible.

In recent era, it is established that semi and unstructured datasets are more valuable than structured. Companies are investing millions of Dollars in technologies and frameworks like Data Lake, Delta Lake, Lakehouse, Data Vault, Business Vault, HDFS, Object Storage, NoSQL databases etc. to make use of Dark Data which of course will not be Dark once in use 😊.

Dormant Data

As mentioned above, Dormant Data is not something that is not useful, it is something that is not useful all the time. For example, customer buying or spending behavior has changed due to the Covid-19 impact. All historical data before Jan 2020 have nothing to do with buying or spending patterns after Jan 2020. Online shopping has increased more than 100%. Use of Debit/ Credit cards has touched its limits. Security measures on online transactions have hit maximum attention and many more behavioral changes. Referring to the same example, the data before Jan 2020 for Marketing, campaigning etc. has become Dormant for most of the organizations.

‘Data that is underutilized is Dormant Data.’

It has become utmost important for a CDO (Chief Data Officer) to have the lowest level of Dark Data in the organization. In other words, the more the Dark Data, the more the cost without any ROI. Time is not far when Dark Data Management will become the most essential and obvious KPI for CDO (Chief Data Officer).