User identity conditions moral wrongness ratings in non-reasoning large language models
TL;DR
It is demonstrated that unintended contextual conditioning via user identity permeates LLM moral evaluations, posing questions for the AI value alignment discourse regarding how to define acceptable bounds for role-based moral divergence.
Abstract
This study adopts a behavioural bottom-up approach to AI value alignment to investigate whether an implicitly conveyed user identity shifts the moral evaluations of large language models (LLMs). Through a structured, multi-turn conversational protocol across 12,000 interactions, we evaluate AI value alignment in two non-reasoning models, gpt-4.1-mini-2025-04-14 and gemini-2.5-flash-lite. Rather than instructing the models to adopt a persona or prompting them with explicit moral stances, the user's professional role is introduced purely through value-neutral reasoning. The models are then asked for wrongness ratings from 0-100 on ten common-morality rules from Gert's moral framework. The results show that moral judgments vary with the user's role across both models. While grave-harm acts like killing exhibit a strong ceiling effect, contestable rule-governed acts demonstrate role-conditioned shifts that mirror the relationship between the user's profession and the act being rated. These findings demonstrate that unintended contextual conditioning via user identity permeates LLM moral evaluations, posing questions for the AI value alignment discourse regarding how to define acceptable bounds for role-based moral divergence. By doing so, the results contribute to reframing the AI value alignment discourse by suggesting future research on dynamic moral bounds rather than static moral principles or rules as frame of reference.