Biases in Human and AI-Assisted Requirements Elicitation for Mental Health Wearables

Abstract

Responsible AI systems for vulnerable populations require requirements elicitation processes that are themselves free from biases. This paper presents an empirical analysis of cognitive and algorithmic biases across two types of requirements sources: eight domain-expert interviews conducted with human therapists and social workers, and two AI-generated interviews produced by large language models (Gemini and Claude). Based on two complementary bias frameworks - (1) Biases in AI Systems and (2) Fallacies in Substituting LLMs for Human Participants - we analyzed the requirements elicitation processes in the context of developing a Responsible AI Framework for Smart Jewelry Anxiety Management System. Our analysis shows that human and AI informants exhibit structurally different AI bias profiles under both frameworks. Under the first framework, the AI informant introduces uniform confirmation bias and systematic cultural invisibility while failing to surface automation, appropriate use, and ethical concerns, while humans are more culturally diverse. Under the second framework, the AI informant exemplifies the average-human fallacy through WEIRD (Western, Educated, Industrialized, Rich, Democratic)-biased outputs, the identity-essentialization fallacy through stereotyped clinical persona responses, and the substitution fallacy through closed-loop self-referentiality, and the human is more inclined to rely on personal experience. Importantly, both AI informants exhibited structurally identical bias profiles despite differing demographic prompts, providing cross-model replication of the core findings Together, the two frameworks converge on a novel phenomenon we term the AI-Human Requirements Elicitation Biases, yielding six actionable RE guidelines for responsible AI requirements elicitation design.