Knowledge Hub

It refers to the staggering amount and variety of data produced by businesses every day. This is largely due to the variety of enterprise software, mobile apps, storage systems, and data formats each company relies on.

We have data in the shape of Video, Audio, Email, PDF, JSON, XML, Text etc. In other words, data is scattered in different formats and unless all formats are not converted in a standardized format, extracting information will also be challenging or time-consuming.

Imagine that you need to complete your taxes, but all your relevant papers are in different drawers, hidden in closets, and stuffed under couch cushions. Now imagine that you have multiple copies of the forms in these places, and some are written in Greek, while others are written in English and Spanish. How will you do your taxes, or clean your house for that matter, when this is the state of things? Unfortunately, this is a problem that is starting to plague companies across the world. This is Data Sprawl.

With the introduction of the big data era, it became impossible to manually manage those.

‘ITOps focuses on the best in practice IT operations.’

A few main areas of focus for ITOps are.

• IT Service Improvement
• IT Service Strategy and Design
• IT Service Transition
• IT Service Operations
• There are many ITOps lifecycles followed by organizations like
• Maintain the HEALTH of the IT Infrastructure
• Visibility of the IT Infrastructure
• Automation of IT Services
• Optimization of IT Services

Data Backup and Mirroring are two terms that often cause confusion due to their similar nature, but they serve different purposes. Though both entail duplicating data, their goals differ. Let's delve into their distinctions.

Let’s decode it…..

Data Backup:

The main purpose of data backup is to safeguard information for future use, regulatory adherence, or data retrieval in the event of loss. Many contemporary databases possess built-in backup capabilities. For instance, in the banking sector of certain countries, regulations stipulate retaining customer and associated data for a minimum of seven years after the customer departs from the bank. Data pertaining to active customers is kept indefinitely.

It is important to note that not all data is necessary for operational or reporting purposes. Decision support systems often demand particular data, housed within active databases such as OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing). The remaining data is backed up using technologies such as tape drives or affordable storage solutions like Object Storage. Backups can be performed on a daily, weekly, monthly, quarterly, semi-annual, or annual basis, and various methods and techniques exist for taking backups.

CRUD and CRAP functions are our daily used processes for data folks, but surely by name most are not aware of these words 😊.

Let’s decode these…..

As we have been using structured data for the last few decades, we know how to create, read, update, and delete. These basic functions which we use in any database are called CRUD data.

Now, the new era, the current era, surfaced where data started to flow-in with high velocity from web logs, social streams, CCTV, sensor data, videos, Facebook, or Tweeter data, mobile, geo-spatial, and so on. All this data has nothing to do with transactional data, so we used to call it CRAP data 😊. But now these have become one of the most valuable assets of any organization. The speed this data is coming, it’s normally not possible to update or delete it so it’s been created, replicate, appended, and process only.

CRUD stands for Create, Read, Update, and Delete

Views are fundamental in the world of data, serving as a foundational concept. Although it may seem basic, understanding views is crucial as we will delve into more advanced topics such as Materialized Views and External Tables.

Let's break it down...

In the realm of databases, we are familiar with tables, structured with columns and rows, each representing a specific dataset post-normalization (a topic we'll cover separately). Examples include tables for employees, departments, and salaries.

Now, imagine the need to combine information from these tables, like obtaining a comprehensive view of a customer's department and salary. To achieve this, we would typically construct a query that involves joining these tables, providing us with a consolidated result.

However, if we find ourselves running this query frequently, perhaps on a daily or monthly basis, creating a "view" becomes advantageous. A view allows us to encapsulate the query and, when needed, execute a simple SELECT command on the view to retrieve the combined results from the employee, department, and salary tables.

Aggregation Functions are used to perform calculations on a set of values to return a single summarized result. These functions are often applied to columns in a database to derive useful insights from the data.

For better understanding:

1. Aggregation Functions in the Data World:

In the realm of data management and analysis, aggregation functions play a crucial role. These functions are specialized tools used to process and summarize data, particularly when dealing with large datasets. Instead of looking at individual data points, aggregation functions allow us to analyze groups of data and distill meaningful information.

2. Performing Calculations on a Set of Values:

Aggregation functions operate on sets of values, which are typically found in columns of a database. Instead of working with individual records, these functions work with groups or entire columns of data. This allows for the calculation of summary statistics, totals, averages, and other aggregated measures.