The Workload Multiverse: How Analytic Decisions Shape Gaze-Based Measures of Mental Workload

Abstract

In Human Factors, eye-tracking is a popular method to assess an operator’s mental state. This study investigated how the approach of data preparation influences findings on mean fixation duration as a potential indicator of mental workload. An existing eye-tracking dataset from an aviation-related task was re-analyzed with three open-access preprocessing pipelines that detect fixations from raw gaze data. First, we found ambiguous correlations of mean fixation durations between the three preprocessing pipelines. Second, the pair-wise correlations increased when participants were excluded more strictly due to missing values in the data set. Third, two of the pipelines revealed an U-shape relationship between fixation duration and mental workload, while the third one produced a more linear pattern. In summary, our findings show that preprocessing decisions impact conclusions about how fixation duration relates to mental workload and emphasize the need for more research that quantifies the reproducibility of findings in Human Factors.