5 min read
Data Warehouse, Data Lake, Data Hub, or Data Platform?
The growth in the amount of data, data sources, and analysis possibilities means there are also multiple ways to store and process data. Terms like Data Lake, Data Hub, Data Warehouse, and Data Platform get thrown around a lot. Research from Gartner has shown that demand for Data Hubs rose by 20% between 2018 and 2019. Interestingly, Gartner noted that more than 25% of customers thought a Data Hub is a Data Lake solution[1]. Gartner's research illustrates how much confusion there is about what these different terms mean. In practice we also notice there's a lot of confusion; how exactly do these terms differ from each other? This blog provides more clarity on the meaning of these terms.
Data Warehouse
A Data Warehouse (DWH) consists of storing information in an integrated way, with the goal of fueling business decisions and analyses. With a data warehouse, you lay the foundation for Business Intelligence (BI) and Analytics. The data in a data warehouse often comes from various sources within, or outside, the organization. Because a data warehouse brings together enormous amounts of data from different data sources (HRM, DMS, ERP), valuable insights can be gained. A data warehouse saves time for companies that collect data on a large scale and ensures uniformity in the definition of business information. If you want an integrated view of your business operations, a data warehouse is essential.
Data Lake
A data lake is a storage location in which large amounts of raw data are stored in their original structure. In terms of content, a Data Lake can contain both structured and unstructured data. The data structure of individual files and how they should be accessed is not known until the data is used. It's important to see a data lake not as a replacement storage system, but as a place where analysis and research can be done with unprecedented freedom thanks to the relatively low cost of storage and the ease of scaling. Data lakes generally form a good foundation for reporting, visualizations, advanced analytics, and machine learning.
(Un)structured data?
Structured: data from databases, CSV, JSON, etc.
Unstructured: email, PDF, documents, video, audio, binary files
Data Hub
A data hub doesn't store data itself, but manages the flow of data between source systems, target systems, and users. With a data hub, you essentially specify exactly what needs to happen with the data. For example, you can link certain sensor information to an automated order system. As much use as possible is made of the power of the source systems, to guarantee optimal performance. A data hub often takes the form of a hub-and-spoke architecture, where systems can distribute data via the Data Hub, instead of via point-to-point integration where every system is connected to every other system it needs to share data with.
A data hub also gives organizations insight to properly interpret data. Because if you understand what you're looking at, it becomes easier to guarantee the correctness of data or adjust it where necessary. You can literally see, down to the column and row level, how datasets are structured. What's more, you always comply with laws and regulations, because you know exactly who has access to which data and where data is stored. The data in a data hub is not necessarily integrated and can contain different levels of detail side by side, unlike a Data Warehouse. Compared to a data lake, a data hub can offer data in various formats. Where data warehouses and data lakes are endpoints for data, a data hub is a junction through which data flows.
Data Platform
A data platform, also known as a data management platform, is an integrated solution that combines the functionalities of a data lake, data warehouse, data hub, and elements of a Business Intelligence (BI) Platform. Without a data platform, a separate tool or set of tools is usually used for each aspect. This creates a complex landscape in which many tools need to be managed to let data flow from source to end user. A data platform centralizes these solutions into one tool, delivering a product that's much more manageable.
Differences
| Data Platform | |||
|---|---|---|---|
| Data Lake | Data Warehouse | Data Hub | |
| Data | Structured, unstructured, Relational / Non-relational |
Structured, Relational | Structured / Unstructured |
| Schema | Schema-on-read | Schema-on-write | Schema-on-write |
| Storage cost | Low-cost storage | Higher-cost storage | Higher-cost storage |
| Data quality | Raw, possibly unmanaged/uncontrolled | Highly managed | Highly managed |
| Users | Data Scientists / Developers | Business Analysts | Various business users |
| Architecture | Centralized | Centralized | Hub-and-spoke |
Schema on read/write?
Schema-on-read: data is stored unchanged
Schema-on-write: data is transformed and stored in a predefined structure
Conclusion
The enormous increase in data sources and volume, and the varying data needs of different users, create significant challenges for BI/IT departments and others involved in data for analytics, artificial intelligence (AI), and BI. Organizations use all kinds of different tools to process and manage data. It doesn't have to be that way. This is precisely why E-mergo chose to enter into a partnership with TimeXtender. The TimeXtender platform offers a coherent data structure for on-premise technology and cloud. This lets you connect to various data sources and catalog, model, move, and document data for analysis and AI purposes.
TimeXtender aims to bring change to the traditional way of BI development by automating repetitive work. Building a traditional data platform involves a lot of repetitive and time-consuming work. With TimeXtender you can make the switch to an integrated data platform that delivers data insights 5 to 10 times faster thanks to automation. This allows you to save as much as 80% on management and develop 70% faster.
Sources
Ruairidh Smith,
Consultant at E-mergo