Knowledge Hub

Encryption is a technique to convert the normal text in the email, text message, databases, data lakes etc. into scrambling format to make it unreadable which is also called 'CipherText'.

‘Encryption and Decryption use to convert and revert data from another format to target confidentiality requirement and use Key to read data.’

This is one of the most known terms in the Data World concerning Security and there is hardly any organization in the World that doesn't use Data Encryption to protect its data from black hawks.

Let's take this opportunity to go one level down.

Which Data to Encrypt

• Data-at-Rest means data sitting in your database, data warehouse, data lake, in-memory in solutions like delta lake, Databricks, snowflake etc. all must be encrypted.
• Data-at-Transit means when data is moving between devices, such as within private networks, through the Internet, or from laptop to thumb drive etc. all must be in encryption mode.
• Data-in-Use means, when data is exposed to end-users which can be in a BI tool or on a website or web portals like Facebook, LinkedIn etc.

Data Encoding and Decoding is a known term but is always confused with Encryption and Decryption.

Just like Encryption and Decryption, Encoding and Decoding convert and revert data from another format using Key.

Importance of Keys

• Encryption and decryption conversion KEYS are used by known entities or individuals.
• Encoding and decoding KEYS are used by unknown entities or individuals i.e., publicly.

Data Encoding and Decoding is normally used by programs or systems internally whereas Encryption and Decryption are used by proprietary expensive software(s).

There are many encoding techniques but major four types are.

1. Analog data to Analog Signals: This technique is used when sender and receiver both are Analog compatible like mic to speakers directly.
2. Analog data to Digital Signals: This technique is used sender is Analog, and receiver is Digital compatible like mic to computer for recording purpose.
3. Digital data to Analog Signals: This technique is used when the sender is Digital, and receiver is Analog compatible like from computer to speakers.
4. Digital data to Digital Signals: This technique is used when sender and receiver both are Digital compatible like data transfer from computer to computer.

Data Subsetting is as simple as taking a small piece of data, with referential integrity in place, for some activity. For example, there is a requirement to build a data science model and it requires data for users aged between 25-35 from X Location having certain buying/ spending patterns. For this requirement, a particular dataset will be extracted for the data science development team to build and train their data models.

Don't confuse Data Subsetting with Data Masking. Data Masking is still required on this smaller piece of the dataset from a security point of view. Having said that, conducting a Data Subsetting exercise is not as simple. One first must understand the functional requirements of the use case, then one must also understand what data is there in the database as whole. Wrongs or partial Data Subsetting will end up producing wrong data model results and will damage trust in data by decision-makers.

This term is normally known by the Security department and is tagged with Hackers.

‘Data Scraping is also known as Web Scraping.’

The minor difference between both terms is Data scraping means searching the location of the data and then extracting it. In other words, it does not copy and paste rather extract data in a precise and accurate manner whereas Web scraping involves extracting exactly everything from web pages.

Data Scraping or Web Scraping is the process of importing information from a website into a spreadsheet or local file saved on your computer.

Popular uses of data scraping include:

• Research for web content/business intelligence.
• Pricing for travel booker sites/price comparison sites
• Finding sales leads/conducting market research by crawling public data sources (e.g., Yell and Twitter)
• Sending product data from an e-commerce site to another online vendor (e.g., Google Shopping)

It allows us to take unstructured, scattered data from multiple sources and collect it in one place and make it structured for better use.

In the current era there are many data stores like Database, Data Warehouse, Data Marts, Big Data, Data Lake, Data Vault, Business Vault, SQL, and NoSQL etc. Use of all these terms is different with their own horizon, their own skill sets whereas tools and languages can be the same for all of these.

First one must understand is that Database is a datastore, we can’t link it to any data modelling solution. In other words, OLTP or Data Warehouse or Data Mart or OLAP all use databases as a medium to store their datasets. Additionally, Databases can be SQL or NoSQL.

Now, OLTP is on one side of the line and Data Warehouse/ Data Marts and Big Data/ Data Lake etc. are on the other side. Normally, databases are called OLTP (Online Transactional Processing) whereas Data Warehouse/ Data Marts fall under the term OLAP (Online Analytical Processing).

As far as Big Data/ Data Lake is concerned, Data World has yet to tag these with either OLTP or OLAP but for sure these are not meant for OLTP. At the same time NoSQL is stepping into OLTP zone and is supporting OLTP as well.

The idea of a Data Warehouse emerged when organizations realized that transactional systems (OLTP), while excellent for recording day-to-day operations, were never designed for heavy reporting or analytical workloads. As business leaders demanded consolidated views of performance, finance, sales, and customer behaviour, data modelers introduced the concept of a centralized analytical datastore, the Data Warehouse, to relieve OLTP systems from the burden of reporting and to provide trusted, historical insight.

It is important to clarify a common source of confusion: a Data Warehouse is not the same as Dimensional Modelling. The warehouse refers to the central repository that consolidates data from multiple source systems, while dimensional modelling is only one possible technique for structuring that repository. A Data Warehouse can be designed in different ways, depending on organizational priorities such as performance, scalability, or agility.

This topic is very close to many experienced data modelers because it is surprisingly misunderstood. The confusion usually begins with one assumption: if both systems use Third Normal Form (3NF), then they must be conceptually the same. They are not.

In data architecture, the Data Warehouse concept popularized by Bill Inmon embraces the use of 3NF. Since 3NF is also widely used in OLTP systems, many architects assume the modeling philosophy and operational behavior are identical. The structure may look similar, but the intent, behavior, and lifecycle of data are fundamentally different.

OLTP 3NF: Designed for Transactional Efficiency

In an OLTP (Online Transaction Processing) system, the objective is operational efficiency. The system must support high-volume, real-time transactions such as order placement, payments, inventory updates, and customer modifications.

The primary goals are:

• Fast inserts, updates, and deletes
• Concurrency control
• Storage optimization
• Data consistency for current state

Immutable Data Warehouse has come from immutable data. Data folks are aware of it but don’t use it more often.

‘Immutable means not changeable.’

Now, the question is how it’s relevant to us in Data World. We know when we are inserting new data in a database, at the same time we also have the option to change it by using update or delete commands which means data is mutable.

Mutable feature is a MUST TO HAVE in any database at a technology level, means, any databases or data store MUST allow to update or delete. One should or should not update or delete, can be a business requirement but cannot be technology limitation.

‘By default, Data Warehouse is an Immutable datastore.’

Immutable feature is not only about, not able to update or delete data, it’s about which data should be immutable, and which should be mutable. For this, one must understand the data.

For example, CUSTOMER dataset contains many types of data e.g., name, date of birth, parents, children, address, contact details, jobs, cars etc. In this dataset, you must have judged which information is immutable and which mutable.

A Logical Data Warehouse (LDW) is not a new storage technology but an architectural approach to delivering a unified data warehouse view without physically consolidating all data into one centralized repository. Unlike a traditional physical data warehouse, which extracts, transforms, and loads data into a dedicated storage layer, a Logical Data Warehouse integrates data virtually. It connects distributed data sources and presents them as a single, consistent analytical environment.

The concept does not replace traditional modeling approaches. A Logical Data Warehouse can still support third normal form structures aligned with the philosophy of Bill Inmon or dimensional modeling approaches such as star and snowflake schemas aligned with Ralph Kimball. The distinction lies not in the modeling technique but in the physical location of the data. In an LDW, data may remain in source systems, data lakes, physical warehouses, business vault layers, delta-based platforms, streaming systems, or APIs. The integration happens logically rather than physically.

The term Big Data emerged in the early 2000s as enterprises began facing unprecedented growth in the amount, speed, and diversity of information being generated. Traditional databases and warehouses, while reliable for structured data, were not designed to handle this new reality. Big Data describes not only the massive scale of data itself but also the ecosystem of technologies, methods, and practices developed to capture, store, and analyze it.

At the heart of Big Data is the concept of the “three Vs”, later extended to four or more:

• Volume: the enormous scale of data generated from transactions, machines, sensors, mobile devices, and social platforms.
• Velocity: the speed at which data arrives, often requiring real-time or near real-time processing instead of traditional batch updates.
• Variety: the diversity of data formats, structured (tables), semi-structured (JSON, XML), and unstructured (text, images, videos, audio, logs).
• Veracity (sometimes added): the need to ensure trustworthiness and quality despite scale and diversity.