A rigorous and theoretically grounded framework for human-AI collaboration is introduced, termed collaborative omnimechanism design, based on a novel and computationally efficient contextual bandit variant of boosting for outcome indistinguishability, which provides a principled characterization of human impressionability and introduces a unified optimization scheme.
Abstract
As AI systems are increasingly deployed in high-stakes domains such as healthcare, effectively integrating human expertise with AI outputs is critical for reliable decision-making. A substantial body of theoretical work studies optimal mechanisms for human-AI collaboration (HAC), focusing on when to involve humans and how to combine their expertise with AI predictions. However, these approaches largely overlook a key factor highlighted by empirical evidence: human judgments are impressionable and can vary depending on how AI outputs are presented. For example, when AI systems defer decisions to humans, revealing or withholding the AI prediction can significantly influence human judgments. Moreover, existing work neglects deployment-time variability, where conditions, particularly the cost of eliciting human input, may differ substantially from those observed during training. To address these challenges, we introduce a rigorous and theoretically grounded framework for human-AI collaboration, termed collaborative omnimechanism design, based on a novel and computationally efficient contextual bandit variant of boosting for outcome indistinguishability. In particular, we reduce a calibration-type outcome indistinguishability condition to weighted calibration, providing a novel time-efficiency result, bypassing the high-dimension curse induced by complicated collaboration methods and outcome space. Our framework provides a principled characterization of human impressionability and introduces a unified optimization scheme that enables robust and adaptable decision-making across diverse query costs and evaluation objectives without retraining. We evaluate the framework in three HAC settings, examining adaptation to query costs and prediction objectives without retraining, and show that its induced policies perform comparably to or better than the evaluated baselines.
Safely overseeing AI is increasingly challenging for the humans doing it, as they must evaluate longer and more complex outputs with every new model generation. To mitigate this challenge, we could leverage complementarity: where humans and models make different mistakes, combining their judgements could yield an overs...
S.-X. Shi, M. Christ, Eric Modesitt et al.· 0 citations
This work investigates feedback interventions targeting overrides of AI advice through a behavioral study and finds that such feedback influences reflection-on-action, while unidirectional feedback risks reducing subsequent reflection-in-action.
Joshua Holstein, P. Spitzer, S. Schäfer et al.· 0 citations
AI alignment is standardly framed as making AI systems track what humans want. But what users want, under everyday conditions, is frequently at odds with what users themselves would reflectively endorse. Across AI and the broader attention economy, the former is systematically elicited and optimized for while the latte...
Jakob Stenseke, Jocelyn Zhu, Dylan Hadfield-Menell· Proceedings of the AAAI/ACM...· 0 citations
The critical but underexplored role of human mental models—specifically, users' understanding of when and where an AI system errs—in shaping the effectiveness of AI-advised decision-making is examined, arguing that practitioners must optimize not only for what the AI gets right, but for how predictably humans can learn...
Jonathan H. Westover· Human Capital Leadership Rev...· 0 citations
This workshop explores how design choices, such as reducing human-like AI design or communicating (un)certainty of responses, can support more informed and balanced use of AI.
S. Diefenbach, D. Ullrich, P. Preuschoff et al.· Adjunct Proceedings of the 1...· 0 citations
As AI systems increasingly interact with people and make decisions about them, understanding human interpretations becomes an important part of developing human-centered AI. Conventional machine learning and AI systems are largely developed under the assumption that a single definitive ground truth exists, with variabi...
Jing-Yao Wu, Mohammad Tariqul Islam, Per Rådberg Nagbøl et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.