Rethinking Accuracy: A Weighted Error-Based Metric for Data Quality
TL;DR
TOMME - an initial approach to a universally applicable weighted error-based metric for data quality that allows the data quality of a dataset to be assessed based on a single score and enables rapid assessment and automated processing for optimization algorithms.
Abstract
Real data often contains errors, which is why data engineers spend a lot of time creating data cleaning pipelines to ensure the best possible data quality. However, it is often difficult to compare the results of different pipelines and decide which pipeline leads to the best results. There are many different metrics that are designed for different use cases, but they often take only a portion of the data into account. There is a lack of universally applicable metrics for measuring data quality that can be used in many different scenarios. That is why in this paper we are presenting TOMME - an initial approach to a universally applicable weighted error-based metric for data quality. This allows the data quality of a dataset to be assessed based on a single score. While a detailed data quality evaluation remains important, the use of a single score enables rapid assessment and automated processing, for example, for optimization algorithms. By using different weights, the score can also be precisely adjusted to the specific use case. That is why we named it TOMME, which stands for"The One Metric Measuring Errors". As the name suggests, it measures errors in the data. It can thus be considered a generalized, weighted form of accuracy.