Getting Started with AutoML: Model Selection

Who wouldn't want to predict the future? With Qlik AutoML, you can easily create predictive models based on your data. As the name of the feature suggests, Qlik AutoML automates a large part of this process for you. Even so, as a user you still have a lot of influence over the effectiveness of the predictive model. In part 2 of this blog series, we explain what you can do to make your predictive models with AutoML even better, and how you can integrate the result into your existing Qlik dashboards.

What is model selection?

In the first part of the blog series we explained how you can apply "feature engineering" to optimize the training data for your model. The next step is to actually have models trained.

You already make the first choice in the very first step of configuring an experiment. Depending on the type of predictive model you want to build, AutoML selects a number of possible prediction methods (algorithms) that can be used for the predictive model. There are 3 types of predictive models that Qlik can apply:

• Binary classification: The field you want to predict has 2 possible outcomes, for example Yes or No, 1 or 0, true or false.
• Multi-class classification: The field you want to predict has more than 2 possible outcomes, but the number of options is limited. For example, if you want to predict which color car someone drives, the options are blue, white, red, green, gray.
• Regression: The field you want to predict is a numerical value. For example, if you want to predict how many flowers will be purchased in a certain month.

Depending on the type of predictive model, there are therefore different algorithm options that can be used. You can choose to include all algorithms in the experiment, but if you already have a preference beforehand, you could exclude specific models.

Once you've chosen the algorithms and indicated which features will be used to train the models, AutoML can get to work. A version is trained for each model. But how do you choose which model you'll ultimately use? In this blog we explain what choices you can make in this regard.

The scoring system

As soon as AutoML has finished training, you get a results table on screen. In that table, one model is always marked as the winner. Depending on the type of predictive model, a different metric is used to determine this winner.

For binary classification, the "F1" score is used. The F1 is a composite score based on the metrics "Precision" and "Recall." Both scores say something about how accurate the model is. To explain this properly, we first need to look at the so-called "Confusion matrix." Suppose we're trying to predict whether an outcome will be 0 or 1. There are then 4 possible outcomes: we predict 1 and it turns out to be 1 ("True positive"), we predict 1 but it turns out to be 0 ("False positive"), we predict 0 and it turns out to be 0 ("True negative"), or we predict 0 and it turns out to be 1 ("False negative"). By putting the different outcomes into a matrix, you can visualize how good a model is at predicting.

The confusion matrix is important information when selecting the right model. In many cases you simply want to predict as accurately as possible, but if, for example, you're building a model for preventive maintenance to predict whether a machine will break down, you'd rather have a false positive than a false negative. With a false positive you perform some extra maintenance and the machine keeps running; with a false negative the machine breaks down anyway, with all the consequences that entails.

The precision metric is calculated by dividing the number of true positives by the total number of predicted positive cases. In other words, this metric indicates "when we predict 1, in what % of cases is it actually 1." Conversely, you calculate the recall metric by dividing the number of true positives by the sum of true positives and false negatives, in other words "when it's actually 1, in what % of cases do we also predict 1." You calculate the F1 score using the formula:

In other words, the F1 score indicates how often the correct answer is predicted.

Now you might wonder: why don't we just divide the number of correctly predicted values by the total number of predictions? This is the so-called "Accuracy." Although accuracy will closely match the F1 score in many cases, this number has an important drawback. Suppose I wanted to build a predictive model that predicts whether an Eredivisie player will make the Dutch national team squad. Most Eredivisie players don't make the squad, so my predictive model would have very high accuracy if it predicts for every player that they won't make it. However, that wouldn't be a very useful model for my purpose. Whether you're better off using accuracy or F1 therefore depends on 2 factors:

• If it doesn't matter whether you predict a false negative or a false positive, and your dataset is balanced (there are roughly as many positives as negatives), use accuracy.
• If there's an imbalance in the dataset, or if false negatives weigh more heavily than false positives, use the F1 score.

Because most real-world scenarios involve imbalanced datasets, the default metric Qlik uses is F1.

Scores for Multi-Class Classification

In the explanation above, we mainly looked at the scoring system for binary classification. For multi-class classification, we use the same methods, but we need to scale them up to the larger number of labels to determine the F1. To determine recall and precision, we need 3 numbers: the number of true positives, the number of positives, and the number of false negatives. Suppose we have a classification problem with 3 labels, A, B, and C. Our confusion matrix might then look like this:

The number of true positives for label A is then still 2 — the number of times we correctly predicted A. The total number of positives for label A is 3; we predicted A a total of 3 times. The number of false negatives for label A is 1, because we predicted C once when it was actually A. This allows us to calculate an F1 score for each label:

To be able to say something about the predictive model, we need to find a way to combine the F1 per label across all labels, so it says something about the entire model. AutoML has 3 different methods for this:

• F1 Macro: Add up all F1 scores and divide by the number of labels.
• F1 Micro: Add up all true positives, and do the same for total positives and false negatives. Use the totals to calculate the F1 with the formula:

• F1 Weighted: Multiply each label's F1 score by the number of actual observations for that label. Divide the total by the total number of observations.

Here too, it again depends on what's important to you for your predictive model. If every label is equally important, choose F1 Macro. If you mainly want to predict the common labels well, choose F1 Micro. F1 Weighted is the middle ground and is especially effective when there's a larger imbalance between the different labels.

Scores for Regression

For a regression model we need to work with a completely different scoring method. Here, we're not trying to predict discrete classes, but rather guessing a number. Imagine we have to guess a number between 0 and 100. The chance we get it exactly right is very small (1% if we limit ourselves to whole numbers), but it makes quite a difference whether we're close to it or far off.

To score the results of a regression model, Qlik uses R2 ("R-squared") as the default score. R2 is a score from -1 to 1 that indicates to what extent the model can explain the change in output (the value we want to predict) based on the various input values.

To explain how we can calculate R2, let's guess a number between 0 and 100, 5 times. Each time we guess a number, we take the difference between the actual number and the number we guessed. We call this the residual. We also take the difference between the actual value and the average of all actual values, the deviation.

As you can see, this sometimes results in a positive number and sometimes a negative number. To make sure all numbers become positive, we square each row. Then we add up all the residuals. This gives us the so-called Sum of Residuals (SSR, Sum of Squared Residuals). We do the same for the deviations, giving us the Sum of Squared Deviations (SST, Sum of Squared Totals). R2 can then be calculated using the formula:

In our example, this results in the following R2:

If we look closely at the formula, we see that R2 essentially looks at how much better or worse our prediction is compared to simply predicting the average value every time. There are 3 possible types of outcomes:

• R2 > 0: the prediction is more accurate than if we always returned the average. The closer to 1 the R2, the better the prediction.
• R2 = 0: the prediction is just as good as if we always returned the average.
• R2 < 0: the prediction is worse than if we always returned the average.

It can happen that you get a high R2 on a training dataset, but as soon as you apply it to a new dataset, you suddenly get a low R2. This can be explained by the phenomenon of "overfitting." With overfitting, the model has established relationships between the input variables and the output that don't actually exist in reality. An example of overfitting is, for instance, when a student prepares for an exam by only looking at old exams from previous years. If a new type of question suddenly appears on a new exam, the student won't know the answer. You can prevent overfitting by using a larger training set or limiting the number of input variables.

A high R2 can be an indication of a good predictive model, but there is a limitation. All deviations are treated as equally important, even though we might find it more annoying to be way off. For example, suppose we need to buy fresh flowers for 5 different sales days. In that case, we'd rather have a predictive model that's a little bit off 5 times than a model that predicts the correct value 4 times but is way off by a large margin the 5th time.

A metric that accounts for this better is the Root Mean Square Error (RMSE). To calculate the RMSE, we again look at the SSR, but this time we divide that sum by the number of observations and then take the square root:

A high RMSE means the predictive model is off by more on average than with a low RMSE. When choosing the right model, it's wise to look at both R2 and RMSE and choose a balance that fits your problem.

Comparing Models

Now that we know how the different models are scored, we can make a good trade-off when selecting the right model. To help you with this, AutoML automatically generates a dashboard showing the different metrics per possible model. You'll find this under the "Compare" tab. The Compare app consists of 2 sheets. The first sheet is called "Model Comparison" and has 2 visualizations. On the left is a bar chart called "Model performance," showing the score for a matching metric for each possible model. On the right is a scatterplot, where you can plot different metrics against each other. Using the selection bars at the bottom, you can select the metric that best fits your question.

The second sheet, called "Details," also contains 2 visualizations. The top table shows all the different metrics that apply to that model for each option. By clicking the column headers, you can sort the table by that metric.

The bottom table shows the "Hyperparameters." These are the settings AutoML used for that specific algorithm. These hyperparameters are especially useful if you have in-depth knowledge of the various algorithms used.

Once you've found the right model for your problem after analysis, press the "Deploy" button to activate the model. You'll learn how this works in more detail in our next blog in this series!

Written by Lennaert van den Brink
Cluster Manager/Senior BI Consultant