Getting started with AutoML: model selection

Who wouldn't want to predict the future? With Qlik AutoML, you can easily create predictive models based on your own data. As the name suggests, Qlik AutoML automates a large part of this process for you. Even so, as a user you still have a lot of influence on how effective the predictive model is. In part 2 of this blog series, we explain what you can do to make your AutoML predictive models even better and how to integrate the results into your existing Qlik dashboards.

What is model selection?

In the first part of this blog series, we explained how you can apply “feature engineering” to optimize the training data for your model. The next step is to actually start training models.

You make the first choice right at the first step of configuring an experiment. Depending on the type of predictive model you want to build, AutoML selects a number of possible prediction methods (algorithms) that can be used for the model. There are 3 types of predictive models that Qlik can apply:

• Binary classification: The field you want to predict has 2 possible outcomes, for example yes or no, 1 or 0, true or false.
• Multiclass classification: The field you want to predict has more than 2 possible outcomes, but the number of options is limited. For example, if you want to predict what color car someone drives, the options are blue, white, red, green and gray.
• Regression: The field you want to predict is a numeric value. For example, if you want to predict how many flowers will be bought in a given month.

So depending on the type of predictive model, different algorithms can be used. You can choose to include all algorithms in the experiment, but if you already have a preference, you can exclude specific models.

Once you have chosen the algorithms and specified which features will be used to train the models, AutoML can get to work. A version is trained for each model. But how do you decide which model you will ultimately use? In this blog, we explain the choices you can make.

The scoring system

As soon as AutoML has finished training, a results table appears. In that table, 1 model is always marked as the winner. Depending on the type of predictive model, a different metric is used to determine this winner.

For binary classification, the “F1” score is used. The F1 is a composite score based on the metrics “Precision” and “Recall”. Both scores say something about how accurate the model is. To explain this properly, we first need to look at the so-called “Confusion matrix”. Suppose we are trying to predict whether an outcome will be 0 or 1. There are then 4 possible outcomes: we predict 1 and it turns out to be 1 (“True positive”), we predict 1 but it turns out to be 0 (“False positive”), we predict 0 and it turns out to be 0 (“True negative”), or we predict 0 and it turns out to be 1 (“False negative”). By placing the different outcomes in a matrix, you can visualize how good a model is at predicting.

The confusion matrix is an important factor in selecting the right model. In many cases you simply want to predict as accurately as possible, but if, for example, you are building a model for preventive maintenance to predict whether a machine will break down, you would rather have a false positive than a false negative. With a false positive, you carry out some extra maintenance and the machine keeps running; with a false negative, the machine breaks down after all, with all the consequences that entails.

Precision is calculated by dividing the number of true positives by the total number of predicted positive cases. In other words, this metric tells you “when we predict 1, in what percentage of cases is it actually 1?”. Conversely, you calculate recall by dividing the number of true positives by the sum of true positives and false negatives, or “when it actually is 1, in what percentage of cases do we also predict 1?”.

You calculate the F1 score with the following formula:

Put differently, the F1 score indicates how often the correct answer is predicted.

You might now wonder: why don't we simply divide the number of correctly predicted values by the total number of predictions? This is known as “Accuracy”. Although accuracy will closely match the F1 score in many cases, this number has one important drawback. Suppose I want to build a predictive model that predicts whether an Eredivisie player will make the Dutch national team squad. Most Eredivisie players don't make the team, so my predictive model will have a very high accuracy if it predicts for every player that they won't make it. However, that isn't a very useful model for my purpose. Whether you are better off using accuracy or F1 therefore depends on 2 factors:

• If it makes no difference whether you predict a false negative or a false positive, and your dataset is balanced (there are roughly as many positives as negatives), use accuracy.
• If the dataset is imbalanced, or if false negatives weigh more heavily than false positives, use the F1 score.

Because most real-world scenarios involve imbalanced datasets, the default metric Qlik uses is the F1.

Scores for multiclass classification

In the explanation above, we mainly looked at the scoring system for binary classification. For multiclass classification we use the same methods, but we need to scale them up to the larger number of labels to determine the F1. To determine recall and precision, we need 3 counts: the number of true positives, the number of positives and the number of false negatives. Suppose we have a classification problem with 3 labels: A, B and C. Our confusion matrix could then look like this:

The number of true positives for label A is still 2, the number of times we correctly predicted A. The total number of positives for label A is 3, because we predicted A 3 times in total. The number of false negatives for label A is 1, because we predicted C once when it was actually A. This allows us to calculate an F1 score for each label:

To say something about the predictive model as a whole, we need a way to combine the F1 per label across all labels. AutoML has 3 different methods for this:

• F1 Macro: Add up all F1 scores and divide the total by the number of labels.
• F1 Micro: Add up all true positives, and do the same for the total positives and false negatives. Use these totals to calculate the F1 with the formula:

• F1 Weighted: Multiply each label's F1 score by the number of actual observations for that label. Divide the result by the total number of observations.

Here too, it depends on what matters most for your predictive model. If every label is equally important, choose F1 Macro. If you mainly want to predict the most common labels well, choose F1 Micro. F1 Weighted is the middle ground and is particularly effective when there is a large imbalance between the labels.

Scores for regression

For a regression model, we need to use a completely different scoring method. After all, we are not trying to predict discrete classes, but guessing a number. Imagine we have to guess a number between 0 and 100. The chance that we get it exactly right is very small (1% if we limit ourselves to whole numbers), but it makes quite a difference whether we are close or way off.

To score the results of a regression model, Qlik uses R2 (“R-squared”) by default. R2 is a score from -1 to 1 and indicates to what extent the model can explain the change in output (the value we want to predict) based on the various input values.

To explain how R2 is calculated, we will guess a number between 0 and 100 five times. Each time we guess, we take the difference between the actual number and the number we guessed. This is called the residual. We also take the difference between the actual value and the mean of all actual values: the deviation.

As you can see, this sometimes produces a positive number and sometimes a negative one. To make all numbers positive, we square each row. We then add up all the residuals. This gives us the Sum of Squared Residuals (SSR). We do the same for the deviations, which gives us the Sum of Squared Totals (SST). R2 can then be calculated with the formula:

In our example, this gives the following R2:

If we look closely at the formula, we see that R2 essentially measures how much better or worse our prediction is than if we simply predicted the mean value every time. There are 3 possible types of outcome:

• R2 > 0: the prediction is more accurate than if we always returned the mean. The closer R2 is to 1, the better the prediction.
• R2 = 0: the prediction is just as good as if we always returned the mean.
• R2 < 0: the prediction is worse than if we always returned the mean.

You may get a high R2 on a training dataset, only to see it suddenly drop when you apply the model to a new dataset. This can be explained by a phenomenon called “overfitting”. With overfitting, the model has found relationships between the input variables and the output that don't actually exist. An example of overfitting is a student who prepares for an exam by only looking at old exams from previous years. If a new exam suddenly contains new types of questions, the student doesn't know the answer. You can prevent overfitting by using a larger training set or by limiting the number of input variables.

A high R2 can indicate a good predictive model, but there is a limitation. All deviations are treated as equally important, even though we may find it particularly annoying when we are far off. Suppose, for example, we need to buy fresh flowers for 5 different sales days. We would rather have a predictive model that is slightly off 5 times than a model that predicts the correct value 4 times but is off by a wide margin the 5th time.

A metric that takes this into account better is the Root Mean Square Error (RMSE). To calculate the RMSE, we look at the SSR again, but this time we divide that sum by the number of observations and then take the square root:

A high RMSE means that, on average, the predictive model is further off than with a low RMSE. When choosing the right model, it is wise to look at both R2 and RMSE and choose a balance that suits your problem.

Comparing models

Now that we know how the different models are scored, we can make a well-informed choice when selecting the right model. To help you with this, AutoML automatically generates a dashboard showing the various metrics for each candidate model. You can find it under the “Compare” tab.

The Compare app consists of 2 sheets. The first sheet is called “Model Comparison” and contains 2 visualizations. On the left is a bar chart called “Model performance”, showing the score for a relevant metric for each candidate model. On the right is a scatter plot in which you can plot different metrics against each other. Using the selection bars at the bottom, you can choose the metric that best fits your question.

The second sheet, called “Details”, also contains 2 visualizations. The top table shows, for each option, all the different metrics that apply to that model. By clicking the column headers, you can sort the table by that metric.

The bottom table shows the “Hyperparameters”. These are the settings AutoML used for that specific algorithm. These hyperparameters are mainly useful if you have in-depth knowledge of the various algorithms used.

Once your analysis has led you to the right model for your problem, click the “Deploy” button to activate the model. You'll learn how the rest of this process works in the next blog in this series!