Statistics Pitfall #4: Unbiased Sample

There are plenty of arguments for letting data guide your decisions. “Numbers don't lie” is a phrase you hear often. At first glance, that's entirely true. Data is an unbiased reflection of reality and a good check on assumptions and gut feelings. Unfortunately, it's not always easy to interpret data correctly. It's no coincidence that universities in the Netherlands, almost without exception, fill part of their curriculum with statistics courses. Anyone who works with data runs the risk of falling into one of many pitfalls. In this series of blogs, we'll explain a number of common pitfalls and give you concrete tips to avoid them. In this fourth and final blog of the series, we discuss the ‘Unbiased Sample’.

The search for an unbiased sample:

These days, we all collect as much data as possible. After all, data is the “new gold” and is supposed to help us make better decisions. In previous blogs, we showed that you need to pay close attention to how you analyze the data to avoid drawing the wrong conclusion. Today we explain that the way you collect the data can also have a major impact on getting the right results.

In 1948, the US presidential election was between Truman and Dewey. It was a tight race, but the newspaper The Chicago Daily Tribune was certain. They had conducted a poll, and victory would go to Republican candidate Dewey. They were so sure of it that they had already printed the June 6 newspaper with the headline “Dewey Defeats Truman” before the official result was known. The result, however, was an overwhelming win for Truman, who triumphantly posed for photographers with the newspaper in hand.

What had gone wrong?

The newspaper had conducted the polls by randomly selecting phone numbers and calling to ask who the person answering the phone planned to vote for. In 1948, however, a home telephone was a luxury item, so it was mainly wealthier, middle-class Americans who had a phone number. As a result, the newspaper's sample had an overrepresentation of Republicans.

Another pitfall in data collection is so-called “survivorship bias.” During the Second World War, the US military conducted research on their combat aircraft. They wanted to reinforce the planes with extra armor. To determine where to place the extra armor, they looked at the bullet holes in aircraft that had returned after being fired upon.

Initially, they decided to place armor on the fuselage and wingtips. After all, that's where the planes were hit most often. Statistician Abraham Wald, however, advised reinforcing the cockpit and engines instead. Why? Because the planes being analyzed had all made it back. The planes whose cockpits or engines had been hit had all gone down and were therefore not part of the set of aircraft in the analysis.

So it's important to think carefully about who or what you're measuring when collecting data, but how you carry out the measurement can also have unintended effects. Suppose, for example, you ask a group of people in a survey which color they find most beautiful: red, green, or yellow? The results are:

You can't then claim that their favorite color is green, only that it's preferred over red and yellow. After all, you gave them a choice between those three options, but colors like blue or orange couldn't be given as an answer. Factors such as the wording of the question, and even the order in which you ask them, can also influence the answers.

In fact, the very act of trying to measure an effect can influence the values you're measuring. That sounds paradoxical, but it's a well-known phenomenon also known as the “Hawthorne Effect”. Hawthorne Works was a factory in Illinois in the 1920s where researchers tried to increase efficiency by making small changes and then studying the effect on productivity. Most changes seemed to have a positive effect during the study itself, but these effects disappeared almost immediately once the study stopped. The analysis showed that employees worked harder as soon as the researchers were “watching them closely.” We often see a similar effect in exit polls at elections, where voters frequently tell the researcher they voted for the “socially desirable” party, while in reality they voted for a completely different one.

4 tips for good samples

Good analysis therefore starts with collecting good data. When working with data, you'll often hear the mantra “garbage in is garbage out.” Here are a few tips to improve the quality of your sample:

#1 Random samples

If you can't measure every transaction, try to work with random samples as much as possible. The more random, the better, so also vary, for example, the moments at which you measure.

#2 Quotas

If you know in advance which factors have a (major) influence on your process, you can apply quotas to your sample. If, for example, gender matters and you know the male-female split in your target audience is 50%-50%, you survey 10 men and 10 women (this is also known as “stratified sampling”).

#3 Stratified sampling

If you've collected enough data, you can also apply stratified sampling afterward by randomly drawing data points from your full dataset that meet the required categories and leaving the rest of the data out of your analysis.

#4 A good sample

The Hawthorne effect mainly occurs when the people being observed feel threatened. For a good sample, it's in your own interest to ensure a socially safe and open atmosphere. Sometimes it's possible to measure without the observed person being aware of it, but be mindful of the ethical and moral objections that can raise.

Written by Lennaert van den Brink
Senior Consultant