Predicting with Qlik AutoML

The holy grail for every company working with data is predicting the future and being able to act on it. It is the highest rung of the Gartner ladder for data analytics maturity. Until recently, Qlik was mainly suited to supporting the first two rungs of that ladder. That changed when Qlik made AutoML available within its Cloud platform in September of this year. With AutoML, we once again see an earlier acquisition, that of Big Squid, return as integrated functionality. In this blog, senior consultant Lennaert explains exactly how Qlik AutoML works.

Gartner ladder

As you can see in the Gartner ladder for data analytics maturity below, there are 4 levels of maturity. The first step is descriptive analytics, which answers the question of what happened. Think, for example, of a simple annual financial statement. It shows exactly how much money came in and how much was spent. One step further, you also want to know why costs were higher this year, for instance. You start looking for relationships and explanations. Predictive analytics is about using historical data to make a prediction about what is going to happen. Think of questions such as: “Is this customer going to cancel their subscription?”, “When will this machine break down, and what exactly will break?” or “How much extra revenue can I expect if I launch an additional discount campaign?”. The final step is to actually dictate what will happen through actions, for example: “If we give this customer a 10% discount on their monthly fee now, they will probably not cancel their subscription”.

Qlik AutoML

With AutoML, you can also take the steps towards predictive and prescriptive analytics. AutoML stands for “Automated Machine Learning”. What normally makes the steps to predictive and prescriptive analytics so difficult is that they generally require highly specialized knowledge, which is hard to find in the market. It is no coincidence that Machine Learning Engineer ranks 4^th in LinkedIn’s top 25 fastest-growing job titles of 2022. So it is not always easy to hire an ML expert, and once they are on board, they will quickly be swamped trying to meet all the demand.

Machine Learning

Qlik AutoML makes it a lot easier to get started with Machine Learning. We explain exactly how it works further on in this blog, but the basic concept is that you give AutoML a dataset to train on and tell it which value you want to predict. Based on the type of prediction, AutoML then trains and optimizes a number of suitable algorithms and tells you which one works best. You can then apply that prediction model to a new dataset, which you can immediately visualize in Qlik. This way, not only can an ML expert get much more work done in less time, but ML also becomes more accessible to data analysts with less specialist expertise.

Getting started with AutoML

The starting point for AutoML is data in the catalog of your Qlik Cloud environment. It does not matter whether this is a QVD or, for example, an Excel or CSV file. The most important thing is that this data file contains at least a “Target” and a number of potential factors (“Features”). The “Target” is the column you want to predict, so in our case the column “Churned”, which indicates with “yes” or “no” whether a specific customer eventually cancelled their subscription. Potential factors in our case include “PlanType”, the type of subscription, or “ CustomerTenure”, the number of days a customer has been a subscriber. To start the experiment, we click “Add new” in Qlik Cloud and choose “New ML experiment”.

Once we have clicked that, we need to give our experiment a name and description and choose the Space where it will be stored. After that, Qlik asks us to select the source dataset containing our historical data with the Target and Features. AutoML will then ask you which column contains the “Target”, in other words, which value you want to predict. As soon as you select it, all other fields are automatically marked as potential features. By ticking checkboxes, you can exclude certain columns if you already know in advance that they will not help. AutoML will also flag warnings for some fields, as in our case for the column “Country”. This column contains only 1 value and will therefore never be useful as a basis for a prediction. AutoML excludes this field automatically and shows the reason for exclusion with a warning icon. Finally, in the column “Data Type” you can specify the correct type. This is mainly relevant for numeric values. Here is why: a number has an inherent order, 3 is smaller than 8. That ordering can matter when making predictions. A higher price or fine will probably contribute to a higher chance of churn. Sometimes, however, a number is merely a category and its order does not really matter. Think, for example, of a factory hall number. The height of the hall number will not carry any predictive value.

Once you have defined the “Target” and “Features” and applied the right categorization, AutoML will itself propose a number of algorithms that could work well for making the prediction. You can choose whether to train all of them or exclude a specific algorithm. You can also indicate whether “Hyperparameter optimization” should be used. It is beyond the scope of this blog to explain what this optimization involves, but in short, with “Hyperparameter optimization” AutoML tries to tune the internal parameters of the algorithms as closely as possible to your specific case. In some cases this can take quite a long time, which is why you can set a maximum time for the optimization. If you set it to 1 hour, for example, AutoML will stop optimizing after that hour (if it had not finished yet) and present the best result achieved so far.

Once these settings are as you want them, press the “Run experiment” button. From that moment, AutoML gets to work training all the selected models on your dataset. Depending on the type of variable you are trying to predict, you will see a slightly different result. In our case, we are working with a binary classification (a customer either cancels their subscription or not). Other types of problems are multiclass classifications and numeric predictions.

As you can see above, in our example “XGBoost Classification” came out on top. By default, Qlik presents a table (1) with a number of commonly used metrics for evaluating such an algorithm. In this default view you see metrics such as “Precision”, “Accuracy” and “Recall”, but under the button at the top right (2) of the table you can find many more metrics. It is beyond the scope of this blog to explain each of the possible metrics, but for the experienced ML expert these metrics describe how good the algorithm is and in what ways it can occasionally still go wrong.

Fortunately, AutoML also visualizes the most important conclusions based on these values. The charts at the bottom left show which features are important for the model and to what extent they influence the target. The charts at the bottom right give an impression of how accurately the model predicts the outcome and which types of errors are made most often (false positive or false negative).

From this point, you can go 2 ways. If you are not yet satisfied with the outcome, you can try a second configuration, for example by including or excluding a certain Feature, or by training the winner again, this time with Hyperparameter optimization switched on. If you do, a new set of algorithms will be trained and the results will be added to the table. That way, you can immediately see whether the changes have a positive or negative effect on the result.

Once you are satisfied with the result, you can select it using the checkbox and then choose “deploy”. Again, you need to give the deployment a name and description and choose a space where the deployment will be stored. Once saved, you can open this deployment. By clicking “Create prediction”, you can then choose a new dataset to which you want to apply the prediction. In our case, that is a dataset of current customers whose churn we want to predict. The output of the prediction is stored in a new dataset, so we need to enter a name for that as well. In addition to the predictions, we can also have a number of other interesting QVDs written out, such as the SHAP values. We also need to specify the key linking these QVDs to our original dataset. Ideally, that is an ID column that already exists, but if not, you can have AutoML generate an ID. Finally, when we click the “Predict” button, Qlik gets to work and creates the new QVDs.

As mentioned, the result is a new dataset containing the prediction. In our case, this gives us a table with 4 columns: the ID as the key field, the column “Churned_predicted”, containing the predicted value, and the columns “Churned_no” and “Churned_yes”, which indicate the predicted probability of each value. This means we can already take action when the probability of churn exceeds a certain percentage. We can then load this dataset into a dashboard to visualize it.

Want to see Qlik AutoML in action?

As you have seen above, it is relatively easy to get from a dataset to a prediction with AutoML, while there is certainly plenty of room for the real ML experts to dig deeper. The feature is currently available free of charge to all Qlik Cloud Enterprise customers, although with a number of limitations. In the free version, for example, you are limited in the size of the dataset you can use to train the model: the number of rows multiplied by the number of columns may not exceed 100,000. You can also only run a limited number of deployments with the free version. Are you interested and would you like to see AutoML live in action? Get in touch for a demo via our contact page.