Statistics pitfall #1: Cherry Picking

There are many arguments for letting data guide your decisions. “Numbers don't lie” is a common saying. At first glance that is completely true. Data is an impartial representation of reality and a good test for assumptions and gut feelings. Unfortunately, it is not always easy to interpret data correctly. It is no coincidence that universities in the Netherlands almost without exception fill part of the curriculum with statistics classes. Anyone who works with data risks falling into one of the many pitfalls. In this blog series we will therefore explain a number of common pitfalls and also give you concrete tips to avoid them. In this first blog we discuss ‘Cherry Picking’.

Cherry Picking:

Perhaps the best-known pitfall, because it is also often deliberately abused. Cherry Picking means that you pick exactly those data points from the set that best support your story and leave out the rest when presenting it to others. A good example is this clip from the Dutch TV show Zondag met Lubach from October 2016.

The Dutch news ran big headlines: “Police checks on ethnic minorities unjustified in 40% of cases”. That sounds pretty serious, but when you looked at the full study, you saw that it concerned only 12 of 29 examined cases in a set of 272 police stops. The researchers in the report themselves concluded: “the quantitative analysis shows that there is no significant correlation between the appearance of citizens (…) and the intervention decision”.

The tricky thing about Cherry Picking is that it is not always deliberate. For example, on a dashboard you may come across a KPI object like this one:

Last month the conversion was 40% and this month 60%. A huge improvement, things are going well! But what if the chart over the past months looks like this?

That 40% last month is a huge dip, this month we are at 60%, but we are still far from the 97% we achieved in the first quarter. So things are not going well at all! We need to get to work as soon as possible to bring the ratio back to its old level. Or maybe not? Because what if we also put last year's figures next to it?

We have achieved a higher conversion than last year in almost every month, and the figures show a similar pattern. So nothing to worry about: we are doing slightly better than last year, but the differences are small.

So you can create a visualization for any conclusion using the same dataset. Pretty dangerous, but what can you do to arm yourself against Cherry Picking? We give you 3 tips:

#1 Ask questions

When someone presents data to you, ask yourself: “What am I not being told? Do I have enough context?”. If you have access to the dataset yourself, it is often useful to explore the data yourself. A dashboard can help you with this.

#2 Transparent filters

Make clear which filtering you have applied when you present data yourself. In Qlik Sense, for example, the selections are always shown at the top of your screen, but in Power BI this is not always the case. If you make an export or report yourself, it is good to note which selections you used.

#3 Definitions

Make sure you have good definitions of KPIs. If you have agreed clearly in advance how performance indicators are to be measured, the room to cherry-pick (consciously or unconsciously) becomes considerably smaller.

More statistics pitfalls?

Has this blog made you curious about the other blogs in this series? Read them all at your leisure via the buttons below.

Pitfall 2Pitfall 3 Pitfall 4