Knowledge Hub

As the name says all, Data Sharing is about making data accessible to all the relevant stakeholders.

‘Data Sharing is about eliminating Dark Data.’

Data Sharing is one of the key components in Data Driven Organization where organizations aim to build data culture with Data Liberalization.

Data Sharing Channels

• Using Business Intelligence tools
• Using languages like SQL, Python etc.
• Using APIs or direction Using JDBC/ ODBC Connectors
• Using built-in Drivers
• Using Machine Learning or Using Data Mining Models
• Exporting in Excel, CSV, FF formats
• Pushing data to other databases and applications
• Letting other databases and application to Pull data.
• ETL/ ELT tools
• Audit, Compliance or Regulatory Reports etc.

Data Aggregation is the most used term in our daily lives whether it’s in office or in home, we use it everywhere.

‘Data Aggregation is about converting details into summarized format.’

For example, if someone asks you how many months there are in a year? You won’t be taking names of all the months rather you will answer, there are 12 months.

Referring to the image, if your manager asks how many installations our team has done, you rather sending left side details, you will send right side summarized report to your manager i.e., in-total there are 6 iOS and 3 Android installations.

As we normally use databases or data stores to store data, there are different aggregation functions normally used to summarize the data.

Aggregation Functions as per ISO Standards

• COUNT
• SUM
• AVG
• MIN
• MAX

Few other Aggregation Functions

• APPROX_COUNT_DISTINCT
• CHECKSUM_AGG
• COUNT_BIG
• GROUPING
• GROUPING_ID
• STDEV
• STDEVP
• STRING_AGG
• VAR, VARP

Canned Data is something which is readily available at one click. A dataset prepared based on business requirements i.e., if required in pre-aggregated format, in one accessible format.

The aim of canned data is to have a dataset that needs no more cleansing, no more wrangling, no more reconciliation etc. Aim is, User to just click it and data should be served.

Imagine Canned food, which are prepared and seals to be used anytime, anywhere, by anyone. Most of the canned food is ready to eat. Same is with Canned data, ready to use. OLAP Cubes are a perfect example of canned data where pre-calculated and pre-aggregated data is stored in a multi-dimensional table, ready to be consumed anytime.

‘Canned data can be refreshed with latest data using same existing data element.’

Canned Reports

Just like Canned Data, Canned Reports has the same approach where static designed reports are built for non-tech savvy stakeholders, who just want to click once and see the expected required output.

Adhoc Reports

Whereas Adhoc reports are for tech savvy users who know how to play with the BI tools and who also have business knowledge of data element shared in the tool. Using Adhoc reporting, users can build their own reports, dashboards by drag and drop exposed tables and columns. Users can slice and dice the reports. Users can drill-in and drill-out.

Tools like

• Oracle Forms, SSRS etc. are considered as Canned Reports generators.
• Tableau, Power BI, Qlik etc. are considered as Adhoc Reports generators.
• OBIEE, Cognos etc. are considered as Semi Canned and Semi Adhoc Reports generators.

Scope Creep is a known term among Project Management but is worth discussing it among data folk as scope creep can also impact data architecture and data modelling.

‘Scope Creep are those new requirements which are never formally added in project scope or if added, are without estimating impact of project deadline.’

Scope Creep means adding new requirements without analyzing impact on time and cost. I personally follow the thumb rule of TCS.

‘TCS: Time, Cost and Scope.’

Whenever 1 of the above mentioned 3 pillars of project management is impacted, the remaining 2 WILL also be impacted. Scope Creep is referred as e.g., a new requirement is added to the project scope, but the Project Manager didn’t estimate how much more effort is required and will there be any additional cost involving it. At the end of the day, it may impact Project Go-Live date which no one wants.

Reasons of Scope Creeps

• Deficiency of a Vibrant and Thorough Scope
• Absence of a Stakeholders Concurrence 
• Weak Project Sponsor Decision Management 
• Contradictory Business Users Views
• Less participation of all stakeholders during Project Lifecycle
• Bad Management of Risk and Issues Proactively
• Crossing Prioritizing Requirements
• Lack of Change Management Process
• Bad Efforts Estimation
• Bad Evaluation of new Requirements
• Missing Stakeholders
• Non-identification of Out-of-Scope earlier in Project Lifecycle
• Vague Acceptance Criteria(s)
• Poor Assumptions Management
• Poor Constraints Management
• No Clear Project Deliverables

In this era, social media has surpassed every medium to generate data. Social media is not only about writing messages, it also about sending audio or video messages and the moment audio and video (with voice) comes into picture, NLP comes into action.

‘NLP is about using techniques to understand the way humans communicate.’

Using NLP, code tries to convert audio into text and vice versa. Using NLP capabilities there are solutions which talk to humans, understand what they want and then reply to them. NLP also assists in performing actions e.g., a human can ask ‘I want a Chill Coke Zero’, the NLP will recognize this request and get a Cold Coke Drink for customer. At the same time, if Chill Coke Zero is out of stock, using Machine Learning and AI, it can reply to customer in Voice format that ‘At the moment, we are out of stock, would you like to go for Chill Pepsi Black’.

Steps of NLP

• Understand
• Analyze
• Manipulate
• General Human Language or Text

Initially, I planned to write two separate topics on Data Strategy and Data Management but as both are huge areas and the aim of this book is not to train people but rather to make sure readers understand the topics, so decided to combine.

Data Strategy is about defining many elements to make sure when a data solution is implemented, it covers everything to host data, to manage data, to protect data, to maintain data, to make use of data etc.

‘Data Management is one of the Key Element of Data Strategy.’

As the image depicts itself, Data Strategy is about defining all the features and capabilities before the execution process starts. Data Strategy is about tools, processes, implementation planning, maintaining, planning etc., based on business or organization requirements. Data Strategy covers business strategy, organizational roles, data architecture and data management. Data Strategy goals are to innovate, cater to business requirements and address risk and regulatory challenges.

To make it easy, first must understand Traditional Data Stake (TDS). Now, what is TDS lolz 😊. TDS is your On-premises solution which can include any kind of Databases or maybe Hadoop environment as well. Whereas MDS is your tools and technologies on Cloud.

‘MDS vs TDS is more over a comparison between Cloud vs On-premises infrastructures.’

In TDS/ On-premises, if you need new hardware, there is a complete procurement cycle involved. If you need 2 CPU of processing at nighttime and 4 CPU in daytime but you end up paying 4 CPU for 24 hours as you have already bought the hardware 😊.

In MDS, when you are on Cloud, any resource is few clicks away. If you need 2 CPU as night, you only pay when you are using those and same for daytime CPU requirement.

Data Backup and Mirroring are two terms that often cause confusion due to their similar nature, but they serve different purposes. Though both entail duplicating data, their goals differ. Let's delve into their distinctions.

Let’s decode it…..

Data Backup:

The main purpose of data backup is to safeguard information for future use, regulatory adherence, or data retrieval in the event of loss. Many contemporary databases possess built-in backup capabilities. For instance, in the banking sector of certain countries, regulations stipulate retaining customer and associated data for a minimum of seven years after the customer departs from the bank. Data pertaining to active customers is kept indefinitely.

It is important to note that not all data is necessary for operational or reporting purposes. Decision support systems often demand particular data, housed within active databases such as OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing). The remaining data is backed up using technologies such as tape drives or affordable storage solutions like Object Storage. Backups can be performed on a daily, weekly, monthly, quarterly, semi-annual, or annual basis, and various methods and techniques exist for taking backups.

Data Integration is the action of combining data to extract meaningful information and business value.

If we become more technical, data integration is about exchanging content among systems and analyzing it to design decision support systems for business requirements.

Terminology

• A Data Integration Layer is a layer which integrates multiple source systems with target systems.
• A Back end is something which shares data and sources called as source system).
• A Connector is the component which connects source and target so they can share data among each other.
• A Front-End is a target system which utilizes the data for making decisions. It can be OLAP or a digital application.
• An Orchestration is a process in which defines when data will travel from source, when it will be transformed in the shape that target can understand and when it will push to target system.

In the previous diagram:

• The Web Application, the Mobile Application, the Front-End A are Front-Ends.
• The ERP, Legacy Tool, and Back-End B serve as back-end systems.
• To connect to ERP, Legacy tool and Back-End B, those can be connected via C1, C2 and C3, respectively.