Knowledge Hub

Data As a Service (DaaS) is very near to SaaS service in Cloud. We know what SaaS is, it’s about pre-built or OOTB solutions available in the Cloud. Solution providers having their solutions in Cloud with API(s) enabled from where they allow customers to connect and pull data on demand who are then billed based on the data they pulled, this service is called DaaS.

For example, a company builds a solution on cloud where it pulls data from Facebook, Amazon, Ali Baba, etc., and using ML Modelling, predict customers buying behavior, customers surfing behavior, customers segmentation, products categorization, customers demographics etc., this analysis and analytics are then sold as DaaS to organizations in need for these details for marketing or sales planning etc.

The main difference between in SaaS and DaaS is, in SaaS, customer use whole vendor solution on Cloud e.g., customer logs in into vendor solution online and start using it whereas in DaaS service, customer can connect or subscribe to pull the data ONLY.

Data Virtualization, Data Federation and Data Consolidation all are discussed together with respect to exposing data to data consumers.

Where Data Virtualization and Data Federation do not host any copy of data in its environments, Data Consolidation on the other hand keeps complete historical copies of all the data coming from all the source systems. In this process sometimes, a complex ETL or ELT takes place to process data in different data models and sometimes one to one data model is maintained.

Yes, I am referring to Data Warehouses, Data Marts, Data Lakes, Delta Lakes, Data Vault, Business Vaults etc., all these data stores fall under the umbrella of Data Consolidation environments.

Data Island has kind of disappeared as integration technologies have matured.

‘Data Island is a datastore(s) which is/ are not accessible.’

In previous eras when network security was a big concern or even before, when network connectivity was not even there, the datastores were considered as Data Island as external integration was not possible.

Let’s try to understand with examples. We all know WIFI has become a norm in our lives and living without it is like living without Air right? In past when we used to visit a friend’s place we used to greet first, now we first ask, ‘What’s your WIFI Password’ lolz 😊?’. But if you are as old as I am, before 2005 WIFI was not a thing. There were ONLY cables which used to connect our desktops or laptops with network, well I still prefer cable. During that era consider a datastore sitting in different city or region then how to connect? In that situation, we had floppy drives for small data transfer (yeah, I won’t explain what is floppy drive lolz 😊, old folks know what I am talking about. Let’s let newbies find out what is floppy drive), and for huge data we used to have big hard disks and sometimes desktops itself were moved to move data from one location to another location. This datastore, which was not connectable with other datastore, was call Data Island.

Data Silos is always confused with Data Island.

Data Islands are those datastores which are not accessible from external world may be due technology barriers, but Data Silos are those which are only accessible for their relevant stakeholders.

There can be a lot of reasons why data is in silo.

• A particular datastore might be hosting confidential data e.g., finance data, audit data etc.
• A particular data which is not readable e.g., semi or unstructured data sets.
• There is Security or Privacy concerns, so data is not shared among another departments.
• The quality of data is so low that it can’t be shared with management as it may cause wrong decisions and may damage the company’s image.
• Sometime, in development environment different database instances are created for different development teams so if one changes data or table’s structures, other development teams are not impacted.

Why should Data Silos not exist?

As much as Data Filtering is simple as much as it’s so important in Data World.

‘Data Filtering is about reducing the number of rows while pull data from a table.’

On Cloud: In Cloud, resources i.e., compute power or RAM are just click away. In fact, now a days these can automatically increase based on your query requirements. So, if you are using Cloud then, STRICKLY, no one is allowed to run any query either in PROD, UAT or Dev environment without any filter, means in a SQL command one must use WHERE clause as each query is attached with a dollar value. In Cloud, filtering becomes THE FACTOR to maintain resources utilization at minimum, considering as mentioned each resource costs a dollar value.

On-Premises: In contrast of Cloud, resources i.e., compute power or RAM in on-premises systems is limited. To increase resources on premises there is a need for a complete buying cycle by procurement which takes months. With static resources, if someone runs a query without a filter, that query can hold onto all the available resources, locking the rest of the queries. This can create delays in providing other datasets to top management, means delays in decision making, means losing business opportunities, means loss and no one like losses right. In nutshell, in on-premises solutions, filter also is very critical. At the same time developers has the liberty to us any kind of query in development environment as size of data is limited and static.

As the name says all, Data Sharing is about making data accessible to all the relevant stakeholders.

‘Data Sharing is about eliminating Dark Data.’

Data Sharing is one of the key components in Data Driven Organization where organizations aim to build data culture with Data Liberalization.

Data Sharing Channels

• Using Business Intelligence tools
• Using languages like SQL, Python etc.
• Using APIs or direction Using JDBC/ ODBC Connectors
• Using built-in Drivers
• Using Machine Learning or Using Data Mining Models
• Exporting in Excel, CSV, FF formats
• Pushing data to other databases and applications
• Letting other databases and application to Pull data.
• ETL/ ELT tools
• Audit, Compliance or Regulatory Reports etc.

To make it easy, first must understand Traditional Data Stake (TDS). Now, what is TDS lolz 😊. TDS is your On-premises solution which can include any kind of Databases or maybe Hadoop environment as well. Whereas MDS is your tools and technologies on Cloud.

‘MDS vs TDS is more over a comparison between Cloud vs On-premises infrastructures.’

In TDS/ On-premises, if you need new hardware, there is a complete procurement cycle involved. If you need 2 CPU of processing at nighttime and 4 CPU in daytime but you end up paying 4 CPU for 24 hours as you have already bought the hardware 😊.

In MDS, when you are on Cloud, any resource is few clicks away. If you need 2 CPU as night, you only pay when you are using those and same for daytime CPU requirement.

Data Integration is the action of combining data to extract meaningful information and business value.

If we become more technical, data integration is about exchanging content among systems and analyzing it to design decision support systems for business requirements.

Terminology

• A Data Integration Layer is a layer which integrates multiple source systems with target systems.
• A Back end is something which shares data and sources called as source system).
• A Connector is the component which connects source and target so they can share data among each other.
• A Front-End is a target system which utilizes the data for making decisions. It can be OLAP or a digital application.
• An Orchestration is a process in which defines when data will travel from source, when it will be transformed in the shape that target can understand and when it will push to target system.

In the previous diagram:

• The Web Application, the Mobile Application, the Front-End A are Front-Ends.
• The ERP, Legacy Tool, and Back-End B serve as back-end systems.
• To connect to ERP, Legacy tool and Back-End B, those can be connected via C1, C2 and C3, respectively.

Many people mistakenly assume that Integration and Interoperability are interchangeable terms, but they are actually quite different. While they work together, their nature and purpose are distinct.

Let’s decode these…..

Integration involves connecting systems and exchanging data to fulfill a range of objectives [Data Integration explained in different topics]. However, when data is shared between systems, it often requires translation to be understood by the receiving party. Despite the successful integration of systems, these translations conflict with the essence of interoperability.

Interoperability, on the other hand, is the ability to understand data without the need for translation. When two or more systems are linked and exchange data seamlessly without requiring translation for comprehension by the receiving parties, it signifies the presence of interoperability. A notable instance of interoperability is evident in the medical domain, where various hospitals can readily interpret key terms and codes within shared data, whereas an average individual may struggle to do so.

Data Source is essentially where information comes from. It can be anything like a list of customer names in a spreadsheet, sales details in a company's database, or even tweets on social media. These sources can vary in size and complexity, ranging from simple text files to complex, large-scale databases.

In Information Systems, Data Sources can vary. Here are some examples:

• Databases: This is one of the most common data sources. It can be a relational database like MySQL, PostgreSQL, or Oracle, where data is organized in tables, or a NoSQL database like MongoDB or Cassandra, which may store data in more flexible formats.
• Business or Enterprise Applications like CRM and ERP Systems: Systems like Salesforce, SAP, or Oracle ERP are used by organizations to manage customer and business operations data. They serve as rich data sources for analytics and business intelligence.
• Spreadsheet Files: Files like Microsoft Excel or Google Sheets are common sources of data, especially in smaller scale or less technical environments. They store data in tabular form and are widely used for various analytical purposes.
• Flat Files: These include CSV (Comma Separated Values) or Text files etc. They are simple text files used to store data in a structured format and are commonly used for data exchange and storage.
• Cloud Storage: Services like Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage. These platforms are used to store large amounts of data in the cloud and are accessible from anywhere.
• Logs: System logs, application logs, web server logs, etc., are used for monitoring, debugging, and security analysis. They provide raw data about the usage and performance of various systems. They can serve as a Data Source for monitoring applications.