Migrating to Qlik Cloud

Anyone who has followed developments at Qlik recently knows that the company is going all-in on Qlik Cloud. That makes it increasingly worthwhile to consider moving your on-premise Qlik environment to Qlik Cloud. Such a move to the cloud does mean, however, that you also need to make sure all the data from your various systems is brought into Qlik Cloud. There are several ways to do this. In this blog we explain which options are available and what the pros and cons of each method are, so that you too can draw up a plan for migrating your Qlik environment.

By hand

The first method is also the simplest. In Qlik Cloud you can manually upload all kinds of data files to the cloud environment. This method is mainly suitable for ad-hoc analyses or for datasets that don't change very often. Think, for example, of an Excel file with annual budgets. If the data in your file changes, you can simply upload the file again to replace the existing one.

There are some limitations to uploading files, though. Not all file formats are supported, for example, although the most common formats are available (see the image below for the formats currently supported). There is also a file size limit. A file can be a maximum of 100GB. That's quite a lot, but files of 6GB or larger will affect the performance of your cloud environment during the upload, because part of the Qlik Engine will be used for it.

Quick and easy

Suitable for self-service users

Data doesn't refresh unless you upload it again yourself

Large files affect performance and take a long time to upload

Manual work, so dependent on individuals

Connecting directly from the cloud

Just as you're used to on-premise, in Qlik Cloud you can create direct connections to a wide range of sources. More than 70 different connectors are available out of the box for all kinds of source systems, such as databases (SQL, MySQL, Oracle, PostgreSQL, etc.), but also platforms such as SAP, Salesforce and Office 365, and applications like Twitter, Facebook, ServiceNow or MailChimp. Generic connections via REST API or OData are also possible this way, which means you can connect to almost any modern SaaS application.

Once you've created such a connection, you can load and process data from these systems to your heart's content. The main drawback of this method is that your source system must be accessible from outside. For most cloud-based and SaaS applications that's not a problem, but a database system you run on-premise, for example, is something you usually want to shield from the outside world as much as possible for cyber security reasons. Of course, you can configure a firewall very specifically to allow only traffic from the Qlik Cloud addresses (see this page for the current IP addresses of the Qlik Cloud platform), but it still means you create a potential attack route for hackers.

Just as you're used to from Qlik Sense on Windows

Many standard connectors available

Easy to create from the Qlik Cloud interface

Less suitable for on-premise databases that aren't externally accessible

Qlik Data Transfer

Qlik Data Transfer is an application you install on a server in your own network. This means the application sits behind your network's firewall and can usually access various data sources without any problems. With Qlik Data Transfer, you can automatically push data files or data from databases to your Qlik Cloud environment. There are a few ways to do this. The first option is to have Qlik Data Transfer monitor a (network) folder. All files added to or changed in this folder are uploaded immediately to a workspace of your choice in the Qlik Cloud environment. The second option is to connect to a database system via ODBC. You can then define a query in Qlik Data Transfer that runs periodically, after which the result is saved as a QVD and uploaded to the Qlik Cloud environment. The final option is to point to a QlikView or Qlik Sense file (.qvw or .qvf). Qlik Data Transfer will then periodically run the load script of this file, save all the tables in the data model as QVDs and upload them to Qlik Cloud.

However, Qlik Data Transfer does have some important drawbacks and limitations. The application is somewhat harder to configure, and the configuration has to be done on an on-premise server. This is certainly not something self-service users can do. In addition, Qlik Data Transfer lets you upload a maximum of 200 tables or files per source, and the resulting files can't be larger than 6GB. For larger databases, Qlik Data Transfer is therefore not an option.

On-premise installation, so within the internal network and only outbound network traffic

Automated file uploads

Not suitable for self-service users

Not suitable for larger files and databases

On-premise installation also means infrastructure costs

Hybrid Qlik environment

In a hybrid Qlik environment, you use both Qlik Sense on Windows and Qlik Cloud. In this case, the on-premise Qlik environment runs on your own infrastructure within your own network and performs all reloads. As soon as the reload is finished, the resulting dashboard is automatically uploaded to the Qlik Cloud environment. End users then consume the dashboard via the Qlik Cloud environment.
This setup is used a lot during the migration from the on-premise environment to Qlik Cloud. The advantage is that you can immediately offer end users all the benefits of Qlik Cloud's new functionality and user experience, while behind the scenes you have all the time you need to carry out the data migrations.

The biggest drawback of this method is that you still have a large part of your Qlik infrastructure on-premise. Among other things, this means a double Qlik licence and costs for all the hardware and its management. On the other hand, one advantage is that you keep an on-premise Qlik environment, so you still have access to NPrinting and other functionality that isn't (yet) fully available in Qlik Cloud

The best of both worlds, on-premise and cloud

Very quick to set up from an existing Qlik Sense on Windows environment

Extra costs for infrastructure and double licences

Data warehouse in the cloud

In the “connecting directly” option, we already mentioned that connecting to databases hosted in the cloud is a good option. One possible way to get your data into Qlik Cloud is therefore to first unlock your on-premise data sources into a cloud-based data warehouse. This becomes the central place where all the data from your systems comes together and is processed, after which Qlik, as well as other end systems, can use this single version of the truth.
There are all kinds of alternatives for setting up such a data warehouse. In any case, you need a provider of a cloud-based data platform, such as Azure, AWS, Snowflake, Databricks or Google BigQuery. When setting up a data warehouse like this, we recommend also using a data warehouse automation tool. At E-mergo we work a lot with TimeXtender in combination with Azure, but recently there has also been the option to do this entirely through Qlik using Qlik Data Integration. Once the data is available in a cloud-based data warehouse, it's simply a matter of creating a direct connection and you can get started with Qlik Cloud.

One central place to bring your data together

Suitable for virtually all types of source system

Extra investment to set up

Qlik Data Gateway

Qlik Data Gateway is the most recent addition to the list of options for bringing your data into Qlik Cloud. With Qlik Data Gateway, you install a piece of software close to your database (for example on the same server). The software then makes sure a secure connection is established with the Qlik Cloud environment. This way, you can connect to this database directly from Qlik Cloud without having to allow inbound traffic.

Unfortunately, Qlik Data Gateway is only available for a limited number of sources (see this link for the current list), but for those sources it is often the best solution. Configuring the Gateway itself will generally need to be done by an administrator, but once the Gateway is set up, self-service users can easily create connections themselves, just as they can with the “connecting directly” option.

On-premise installation, so within the internal network and only outbound network traffic

Automated file uploads

Suitable for self-service users once the gateway is configured

On-premise installation means extra load on the infrastructure

Conclusion

In this blog, we've shown you that there are many different ways to bring your data into Qlik Cloud. Each option has its pros and cons, so it's important to look carefully at the requirements and wishes of your specific situation to make the right choice. In practice, you'll see that multiple methods are used to upload data within a single Qlik Cloud environment. Would you like to bring your data into Qlik Cloud too, but are you finding it difficult to choose the right option? At E-mergo, we're of course happy to think along with you!