Knowledge Hub

Foresight and Futurology are used interchangeably. And yes you are right this new term is not something we have never encountered before.

Let’s decode it…..

Data: Refers to information, facts, or statistics collected for analysis, reference, or computation.

Futurology: It's the study or soothsaying of possible futures. It involves examining current trends and making prognostications about unborn developments.

Combining these," Data Futurology" suggests concentrating on the future of data, in terms of how data is collected, reused, anatomized, and its impact on colorful aspects of society or technology.

Let go little deeper about conception of" Data Futurology," which involves the study and soothsaying of possible futures related to data.

Let's break it down:

• Study or Forecasting of Possible Futures
• This refers to the practice of examining current trends and making predictions about future developments. It involves analyzing data and information to anticipate how various aspects of a particular field might evolve over time.
• Data Futurology
• This term specifically focuses on the future of data. It encompasses the entire data lifecycle, including how data is collected, processed, and analyzed. Additionally, it explores the potential impact of data on different aspects of society and technology.

Data Source is essentially where information comes from. It can be anything like a list of customer names in a spreadsheet, sales details in a company's database, or even tweets on social media. These sources can vary in size and complexity, ranging from simple text files to complex, large-scale databases.

In Information Systems, Data Sources can vary. Here are some examples:

• Databases: This is one of the most common data sources. It can be a relational database like MySQL, PostgreSQL, or Oracle, where data is organized in tables, or a NoSQL database like MongoDB or Cassandra, which may store data in more flexible formats.
• Business or Enterprise Applications like CRM and ERP Systems: Systems like Salesforce, SAP, or Oracle ERP are used by organizations to manage customer and business operations data. They serve as rich data sources for analytics and business intelligence.
• Spreadsheet Files: Files like Microsoft Excel or Google Sheets are common sources of data, especially in smaller scale or less technical environments. They store data in tabular form and are widely used for various analytical purposes.
• Flat Files: These include CSV (Comma Separated Values) or Text files etc. They are simple text files used to store data in a structured format and are commonly used for data exchange and storage.
• Cloud Storage: Services like Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage. These platforms are used to store large amounts of data in the cloud and are accessible from anywhere.
• Logs: System logs, application logs, web server logs, etc., are used for monitoring, debugging, and security analysis. They provide raw data about the usage and performance of various systems. They can serve as a Data Source for monitoring applications.

Batch Processing is a method of handling data in large quantities by grouping it into sets, or batches, and processing those sets together at scheduled intervals. Unlike real-time processing, where data is handled immediately, batch processing involves accumulating data over a period and then processing it as a group.

Think of batch processing like doing your homework assignments at the end of the day rather than right after each class. You collect all the tasks and work on them together. Similarly, in batch processing, data is gathered over time, and then, as a batch, it's processed all at once.

Few examples for better understanding.

• Grade Processing in Schools: Teachers collect students' assignments, quizzes, and tests over a week and then enter grades into the system at the end of the grading period. For example, instead of entering each student's grade immediately after they submit an assignment, the teacher waits and processes grades for the entire class at once.
• Bank Transactions: Banks often update account balances and transactions overnight, processing all the transactions that occurred during the day in a batch. For example, if you deposit money or make a purchase, the transaction might not immediately reflect in your account balance; it's updated in a nightly batch.
• Payroll Processing: Companies often process payroll in batches, calculating and distributing salaries at specific intervals. For example, employee work hours, overtime, and other payroll-related information are collected over a pay period and processed together to generate paychecks.
• Inventory Management: Retailers may update their inventory levels in batches rather than immediately after each sale. For example, Sales data is collected throughout the day, and inventory levels are adjusted during a nightly batch process.
• Billing and Invoicing: Utility companies may process billing information in batches, generating bills for all customers at specific intervals. For example, Instead of generating a bill for each customer as soon as they use a service, bills are created in a batch at the end of the billing cycle.
• Batch Photo Editing: When editing photos, a batch processing approach may be used to apply the same set of edits to multiple photos simultaneously. For example, if you want to resize, crop, or apply a filter to a set of pictures, you can process them all together in a batch.
• Report Generation: Organizations often generate reports based on collected data at specific intervals, such as daily or monthly. For example, Sales reports, customer analytics, or website traffic reports may be generated in batches to provide a comprehensive overview.

Implementing a robust cybersecurity strategy involves encryption, regular audits, access controls, employee training, and staying updated on emerging threats to safeguard sensitive information effectively.

Few pointers:

• Data Encryption: Employ end-to-end encryption to protect data during transmission and at rest. Utilize strong encryption algorithms to secure sensitive information.
• Access Controls: Implement strict access controls, granting users only the permissions necessary for their roles. Regularly review and update access privileges to ensure they align with current job responsibilities.
• Regular Audits: Conduct periodic security audits to identify vulnerabilities and weaknesses. Regularly assess and update security policies based on audit findings to address new threats.
• Employee Training: Provide comprehensive cybersecurity training for employees to raise awareness about potential threats. Promote a culture of security consciousness to minimize human-related risks, such as phishing attacks.
• Incident Response Plan: Develop and regularly test an incident response plan to respond to security incidents swiftly and effectively. Clearly define roles and responsibilities to minimize downtime during an incident.
• Data Backups: Establish regular and secure data backup procedures to ensure quick recovery in case of data loss or a security breach. Store backups in a remote location to save you from being compromised.
• Network Security: Employ firewalls, intrusion detection and prevention systems to monitor and protect the network. Regularly update and patch software to address known vulnerabilities.
• Vendor Security Assessment: Assess and monitor the cybersecurity measures of third-party vendors handling your data. Ensure vendors comply with security standards and protocols.
• Security Patching: Regularly apply security patches and updates to all systems and software to address known vulnerabilities. Monitor for vulnerabilities that may require immediate attention.
• Data Lifecycle Management: Implement data retention and destruction policies to minimize the risk of unauthorized access. Clearly define how data is collected, processed, stored, and disposed of throughout its lifecycle.

POC stands for Proof of Concept. POC normally follows Sales Context of the product.

In POC, output is most important, not technology. In POC, the implementers make sure output is fulfilling customer requirements and at the same time convince business that this is the solution to go with.

The Goal of POC is to make sure whether Idea is Feasible or not.

‘POC is built and run for Business Users.’

Protype is the next stage of POC. In POC, one has test that yes Idea is Feasible, and it will benefit business agenda.

Now, in Prototyping, tools and technologies comes into picture where one starts converting Ideas in Reality, where it will show how the end-result will look like. It will show, how product will look like. It will show the real end solution.

Prototype is not built considering that business users will jump in and start using it. Prototypes are normally for technical folks to evaluate the end results with dumpy but relevant data.

’Prototype is built and run for Technical Users.’

New name of Pilot is MVP 😊. MVP is the next stage of Prototyping where Tools and Technologies are tested, and end solution is tested by technical folks. Now in MVP, subset of actual requirements is considered, subset of real data is used, subset of hardware and infrastructure is used, subset of software is used, and limited number of features are implemented.

Once MVP is ready, it’s released to actual users to start using it. This gives business users a taste of the actual solution. The biggest advantage of this approach is, with small implementation, with small investment, business can taste the blood and take a decision to go to the next stage or drop the solution.

Now a days, many organizations divide complete solution into multiple MVP(s) i.e., by implementing solutions in chunks, this is also supported by Agile approach where product gets ready in shorter timeframes and then keeps maturing with time. In MVP, just like actual implementation, three environments are spin i.e., DEV, UAT and PROD, along with all the steps like DevOps, CI/CD, Metadata Management, Data Quality, Data Governance, Data Lineage, Master and Reference Data Management, and many other steps which are required to go to Production.

CRUD and CRAP functions are our daily used processes for data folks, but surely by name most are not aware of these words 😊.

Let’s decode these…..

As we have been using structured data for the last few decades, we know how to create, read, update, and delete. These basic functions which we use in any database are called CRUD data.

Now, the new era, the current era, surfaced where data started to flow-in with high velocity from web logs, social streams, CCTV, sensor data, videos, Facebook, or Tweeter data, mobile, geo-spatial, and so on. All this data has nothing to do with transactional data, so we used to call it CRAP data 😊. But now these have become one of the most valuable assets of any organization. The speed this data is coming, it’s normally not possible to update or delete it so it’s been created, replicate, appended, and process only.

CRUD stands for Create, Read, Update, and Delete

Generative Artificial Intelligence (AI) has become a prominent trend in recent times, particularly emphasized by our enthusiastic Sales folks 😊. But let's pause for a moment, are you familiar with the concept of Synthetic Data? Surely, it's not new to you 😊. And for those with a tech-savvy mindset, the term "Generative AI" likely triggers thoughts of Generative Models or Statistical Models, perhaps with a knowing smile.

Indeed, the application of generative and statistical models has been around for decades. In fact, the idea of Generative AI isn’t new at all, it dates back to as early as 1950, when Alan Turing proposed machines that could exhibit intelligent behavior.

Today, however, we're witnessing a resurgence of interest, largely due to the accessibility and power of tools like OpenAI’s ChatGPT, Google Gemini, Anthropic’s Claude, Meta's LLaMA, and Stability AI's Stable Diffusion. These tools represent a leap forward in putting generative capabilities into the hands of everyday users, generating text, images, audio, and even code with natural ease.

In the data world, "Digitalization" and "Digitization" are related concepts, but they have distinct meanings and implications.

Let's decode the difference between the two.

Digitization: It refers to the process of converting analog information into digital format. It involves taking physical or analog data, such as text, images, or sound, and transforming it into digital form, typically represented as binary code (0s and 1s) that computers can understand. For example, scanning a paper document and converting it into a digital PDF file is an example of digitization. The original document exists in analog (paper) form, and digitization converts it into a digital representation.

Digitalization: It is a broader concept that involves the use of digital technologies to transform business processes, activities, or models. It goes beyond converting analog data to digital format and encompasses the integration of digital technologies into various aspects of an organization or system to enhance efficiency, functionality, or value.