7 min read
Data integration for low-code apps
A data-driven organization starts with a solid foundation. This foundation consists of the right tools and knowledge to process data quickly and make it available to users. Data integration and data management have always been important for Data Analytics, so that the right data is available in the right form and at the right time to make important decisions. Today, however, a data-driven organization means more than clear reports and smart dashboards. With low-code software development, organizations are able to digitize business processes in a short time and bring new services to market with the help of innovative and purpose-built apps. Far-reaching digital transformation is essential for many companies' survival and starts with a solid data foundation. In this blog, you'll read why data integration is so important for the success of low-code applications and how you can achieve this.
Low-code software development
Low-code software development has grown enormously in popularity in recent years. The market for low-code platforms has existed for quite some time, and the tools have since developed into mature products. New web applications and mobile apps can be delivered much faster with low-code (in weeks instead of months) compared to traditional development methods. This is partly because the apps are developed visually, using building blocks and drag-and-drop functionality to model user interfaces, functionality, and workflows instead of manually programming them.
Much of the complexity involved in software development is taken off the hands of the developers working with low-code platforms. As a result, in many cases you don't need programming knowledge to get started with low-code and to develop (components of) applications. This gives rise to a new generation of developers, also known as citizen developers. By working with a low-code platform, an organization needs much less IT support for developing, testing, deploying, and monitoring applications. The looming shortage of IT staff is a driving force behind the rapidly growing adoption of low-code.
Leading research institutes also recognize the power of low-code platforms. For instance, Gartner expects that within two years, more than half of large organizations will use low-code development tools to develop applications, and the number of citizen developers involved in app development will be four times greater than the number of professional programmers.
It's clear that organizations can use low-code to realize their digitalization and innovation objectives. Low-code software development is the way for developers and business users to collaborate and build innovative solutions faster. This removes an important bottleneck for innovation, namely the availability of IT staff. But there's another important point of attention that can stand in the way of the success and speed of digital transformation, namely data.
The importance of data integration
Data plays a crucial role in every digital innovation. From experience, we know that part of the time we spend on application development is lost on data preparation. Data needed for an application sometimes comes from different systems and must first be identified, analyzed, interpreted, integrated, and transformed before it can be used in an application. The time spent on this can add up to as much as 50 percent.
A modern data management platform can offer the solution here. By bringing data from various source systems together in a data warehouse in a consistent and automated way, the data can be managed from a central location and made available to users and applications. From experience we know that developing and maintaining a data warehouse costs a lot of time and money, but with modern data warehouse automation tools such as TimeXtender and Qlik Data Integration, this is no longer the case. With these tools you can model the data flows in a graphical interface, while the databases, tables, and ETL scripts are automatically generated. On top of that, metadata and data lineage (the origin of data) are automatically visible, and documentation can be generated from the tools. This makes it possible to achieve an enormous acceleration and quality improvement in the development of a data warehouse.
With the help of data warehouse automation tools, new sources can be unlocked quickly and datasets made available quickly. Application developers can then get straight to work with the data, because they know where the data is located, because the data is available in the right form, and because the data comes with the right metadata to understand its content.
At E-mergo, we've applied the combination of low-code and data warehouse automation in practice in the development of the E-mergo Customer Portal, a web application we recently developed using Mendix. Customers and employees of E-mergo have access to personalized information in the customer portal. For a customer, this means they have direct insight in the portal into ongoing projects, invoices, and open tickets they've submitted to the support department. The information in the portal is retrieved by Mendix from E-mergo's data warehouse, which was developed with TimeXtender. TimeXtender ensures that the relevant datasets from various databases and SaaS applications are combined in the data warehouse.
Data integration in low-code platforms
E-mergo builds impactful low-code solutions with Mendix and Power Apps. The vendors of these tools have since also recognized the importance of data integration. We see this reflected in new features recently added to Mendix and Power Apps that focus on streamlining data integration and dataset reuse.
Mendix Data Hub
Mendix introduced the Data Hub last year during the Mendix World 2019 conference. This is an additional module that adds a kind of data virtualization layer to the development platform. In the Data Hub Catalog, as a developer you can look up and explore datasets and retrieve metadata. The datasets in the catalog are OData services that can easily be added while developing new apps in Mendix Studio Pro. New datasets can be automatically added to the catalog as soon as a developer creates an entity in an app. In addition, datasets can be added manually to the catalog, and there's a Data Hub API. This allows new services to be registered in the catalog for datasets from source systems not yet used in Mendix. Besides finding, registering, and consuming datasets, Mendix Data Hub offers various management functions for implementing data management. Think of setting up data governance, analyzing data lineage, and monitoring data quality. With Mendix Data Hub, it becomes easier to share datasets between applications, and the development of new applications can take place faster as a result.
Mendix Data Hub
Microsoft Dataverse
Microsoft introduced Dataverse for Power Apps last year. Dataverse is the new name for Common Data Service, the data storage layer for Microsoft's Power Apps and Dynamics 365. It's a SaaS storage service in the Azure cloud, which can be used to store the datasets for low-code apps. Dataverse first of all contains a basic set of standard tables in which business information from Dynamics 365 is stored. In addition, Dataverse offers various connectors and features to create your own tables, by automatically reading in and combining datasets from various file formats and source systems. Just like the Mendix Data Hub, in Dataverse you can set up a governance structure to secure datasets while also promoting the sharing of datasets between applications.
Microsoft Dataverse for the Power Platform
Conclusion
Low-code platforms have developed into very mature products with functionality that goes beyond just building apps. Data integration and data management are receiving increasing attention within these tools. This is an important development, because a solid data foundation is essential to deliver on the promise of low-code. Without this foundation, innovation projects cannot be realized on time and within budget. For organizations with ambitions in the field of digitalization, it's therefore important to first pay attention to the data foundation. Only then can application development begin.
Written by Steven Samuels Brusse,
Senior Consultant