Data quality: measuring and improving

Data is the new gold, you've heard people say for years now. But with gold, we know its value varies with its purity. 22-karat gold is worth about three times more than 8-karat gold. But how many "karats" is your dataset? How do you measure data quality? And what can you do to improve the quality of your data? We'll explain it in the blog below!

Why is data quality important?

Before we explain how you can measure data quality, we first need to consider why we actually find data quality important. High-quality data is more valuable because it better describes what reality looks like and is easier to use for various applications. A commonly heard phrase is "garbage in, garbage out," meaning that if you use poor-quality data, you'll get poor-quality dashboards and insights. It's no coincidence that data quality is one of the top trends in the BARC Data, BI & Analytics survey of 2022.

How do you measure data quality?

But what exactly is data quality? Or rather, how do we measure data quality. A common mistake is to equate data quality with data accuracy. For example, we know that the first name of our king is "Willem Alexander," but if we see "Willem Alexia" or "Wellim Elaxendar" in a database, we know that value is incorrect. Accuracy, however, is just one of the dimensions that make up data quality. Different bodies and data quality frameworks each have their own set of dimensions, but in general, data quality is defined as the degree to which the dimensions of data meet their requirements. The tricky part of this definition is that those requirements can differ, and not all dimensions are equally relevant for every company or process. The international body DAMA (Data Management Association) defines as many as 60 dimensions of data quality.

Measuring data quality therefore starts with selecting the dimensions of data quality relevant to you. DAMA has a 4-step plan for this:

1. Determine which of the 60 dimensions apply to the data whose quality you want to measure. For example, if you're dealing with Master Data (often dimensional data such as locations, products, or people), different dimensions apply than when you're working with transactional data.
2. Next, determine the added value of the dimension for your company or process. Measuring, monitoring, and improving data quality costs time and money, so the added value needs to justify this. For example, the degree to which a record meets an industry standard may be very important for one company, while for another there may not even be an industry standard. In that latter case, that dimension isn't of added value.
3. Prioritize the chosen dimensions based on cost-benefit. This way you can start first with the data quality issues that have the biggest impact and keep the budget under control.
4. Establish the KPI and measurement method for the dimensions you want to monitor and improve. Also make sure right away that you jointly determine the target value for this KPI.

Primary dimensions

Now, of course, it's good to apply the above to your own situation and choose the right dimensions, but to give you a bit more of a sense of what these dimensions actually involve, let's look below at the 6 so-called "primary dimensions." These dimensions apply to virtually any dataset and are a good foundation to start with.

Accuracy

We already mentioned accuracy above; it's the degree to which data matches the true value. Examples of errors that can be made here are:

• A person's name is spelled incorrectly, for example Janssen instead of Jansen
• The unit of the entry is incorrect, for example the value 200 for the weight of a bag of candy in a column where the value should be entered in kilograms (instead of grams)

Completeness

Completeness is the degree to which all values of records are actually present in a dataset. This dimension can go wrong in two directions. Values can be missing, but it's also possible that values are present that don't belong in the dataset. Examples of errors made can be:

• Not all products being sold are listed in the products table
• There are people in the customers table who aren't customers
• For some products the color column hasn't been filled in

Consistency

The degree to which a value in two datasets, or within a dataset at different points in time, matches specific rules is what we call consistency. Examples of (in)consistency are:

• An address is registered in Amsterdam, but with Germany as the country
• A customer has one address from January 1, 2019 to March 31, 2022 and the second address starting January 1, 2022

Timeliness

A dataset is timely when it's available within the deadline set for it. For example:

• Yesterday's orders need to be available by 7:00 AM at the latest, but they weren't viewable until 7:30 AM

Uniqueness

In principle, every value in a dataset should only occur once. When a value occurs multiple times, there's a duplicate. This can happen in different ways:

• The entire record occurs in full 2 times:

• The same value occurs with 2 different keys:

• The same key occurs 2 times with different values:

Validity

A value is valid when it meets the rules that apply to that type of value. Note that a value can be valid but not accurate or complete. There are several ways in which a value can be invalid:

• An email address without an @ in it
• A Dutch mobile number that doesn't consist of 2+8 digits
• A date of birth in the year 2202

Improving data quality

Once you've chosen a set of dimensions and defined the measurement values (KPIs) for them, including the corresponding target values, you can start systematically monitoring and improving data quality. When you start this process, you may quickly discover all kinds of issues in your current datasets that cause a low score on your data quality KPIs. It's then a matter of systematically resolving these issues. For this, it's best to use the "leaking faucet" method. Here's how it works:

Suppose you walk into the bathroom at home and suddenly see a big puddle of water. You have a leak! The first thing you do is track down the cause of the leak. You quickly see that someone didn't turn off the faucet, causing the sink to overflow. You could start mopping now, but that's pointless, because water keeps flowing from the faucet. So you first have to turn off the faucet. After turning off the faucet and also mopping up the water from the floor, you can walk away, but the risk is that someone else will come along and forget to turn off the faucet again, so you decide to think about preventive measures. By installing an overflow drain in the sink, you prevent the leak from ever happening again.

We can apply the situation above to a data quality leak as well. Suppose, for example, that in a database you come across all kinds of different variants of a bottle of "water 500ml": "water," "water 50cl," "water spa blue," "waater," etc. This is an example of an accuracy error we want to fix. Just like with the leaking faucet, we don't start by writing queries to change all those different values to the desired "water 500ml," because as long as water is being sold, the incorrect values will keep coming in. Instead, we walk over to the cashier and ask her to always enter the correct value from now on. Then we can run our query to correct the incorrect values. Finally, we go to our IT department and ask them to change the free-text entry field into a dropdown menu where the cashier can only choose from the prescribed values. This way we prevent her from still making a spelling mistake, or a new cashier making the same mistake, and improved data quality is guaranteed.

E-mergo helps you further

In this blog, we've explained the basic principles of data quality. It's good to realize that good data quality isn't a standalone project, but a continuous process. Ensuring good quality requires an interplay of people, processes, and technologies. E-mergo can help you in this process in several ways:

• With Low-code applications built in Mendix or Power Apps, we ensure apps that guarantee high-quality input.
• In TimeXtender, data quality checks can be defined that are automatically executed when new data enters the system.
• With Qlik Data Integration, we can process data in real time through Change Data Capture, so you always have timely data available.
• Dashboards in Power BI or Qlik can give you insight into your data quality KPIs at a glance, so you can immediately tackle new data issues.
• With data strategy workshops, we can help define the right processes to continuously monitor and improve your data quality, and together determine the next steps for your organization.

Written by Lennaert van den Brink
Senior Consultant