JANUS: Evaluating Goal-Conditioned Distortion in Factually Grounded LLM Communication

Abstract

LLM deception is often evaluated through fabricated claims, explicit lies, or strategic concealment. Yet misleading communication can also arise from selectively presenting true facts, for example, by omitting adverse evidence, softening unfavorable details, emphasizing favorable information, or replacing precise qualifications with vague language. Existing benchmarks largely overlook this subtler failure mode. We introduce JANUS, a benchmark for measuring goal-conditioned pragmatic distortion in fact-grounded LLM outputs. Each of its 160 scenarios, spanning 8 domains, provides a fixed pool of favorable and adverse facts and contrasts a neutral decision support with institutional conditioning, such as increasing adoption, or support despite potential harm to affected individuals or groups. By constraining outputs to the same fact pool, JANUS isolates misleading net impressions from hallucination and fabrication. Experiments across 12 LLMs reveal consistent goal-conditioned distortions, showing that current models remain sensitive to incentives and framing and lack robust safeguards against selectively misleading communication. We publicly release our corpus and code. https://github.com/PolydorosG/JANUS