When Labels Lie: Dissociating Source–Scale Bias From Experienced Performance in Decision Support
TL;DR
Examining whether displayed source and scale labels bias DSS use in a timed object-detection task suggested that source and scale labels shape initial selection, trial-level reliance, and post-use reliability judgments in different ways.
Abstract
Decision support systems (DSS) are often evaluated before users have much direct experience with them. As a result, the way these systems are labeled may shape selection, reliance, and perceived reliability from the outset. The present work examined whether displayed source and scale labels bias DSS use in a timed object-detection task. Participants searched cluttered images for a bat after choosing between advisors differentiated by source, either human-crowd or machine-generated, and scale, either medium or large. Two experiments were conducted: a default-label (DL) experiment, in which displayed labels matched the underlying recommendation source, and a flipped-label (FL) experiment, in which displayed source labels were decoupled from the underlying source. Overall, source and scale labels affected different stages of DSS use. Participants selected larger-scale labels above chance, while machine-labeled assistance was not selected more often than human-crowd-labeled assistance. Compliance was high and closely matched advisor accuracy, with slightly higher compliance for large-labeled than medium-labeled assistance. Response times mainly reflected repeated exposure. Perceived reliability ratings showed a significant label condition × displayed source interaction, and error-type analyses suggested greater vulnerability to false-positive recommendations under a human-crowd label. Together, these results suggest that source and scale labels shape initial selection, trial-level reliance, and post-use reliability judgments in different ways.