Skip to content
Book Open access

LLM-Judge Behavioral Coding Scheme for Agent Teammate Quality

Nov 2026 · Proceedings of the 14th International Conference on Human-Agent Interaction · 0 citations · 21 references

Abstract

Evaluating whether an AI agent behaves like a good teammate is becoming a bottleneck as multi-agent language-model systems proliferate, because human ratings do not scale to the transcripts these systems generate. We ask whether a language model can stand in for a human rater of teammate quality, offering a controlled proof-of-concept validation rather than a large-scale study. Drawing on the autonomous-agent teammate-likeness construct from the human-autonomy-teaming literature, we define a six-dimension behavioral coding scheme covering altruistic, benevolent, interdependent, emotive, communicative, and synchronized conduct, adapted for a language-model judge. We validate it by rewriting the partner’s lines in team-task transcripts as cooperative, neutral, or defective while holding the rest fixed, and having five judges from five model families rate every version. Every judge orders the manipulated behavior as intended and every pair agrees on relative ordering, though absolute calibration differs by up to about a scale point. On registers matched for length and politeness the judges separate cooperative from degraded conduct but resolve subtle defection only weakly; a blinded single-rater human pilot reproduces both the orderings and that limit. The scheme is therefore valid for relative comparisons but not for absolute thresholds applied across judges. We give design implications for LLM-judge teaming evaluation, and release the scheme, stimuli, prompts, and ratings.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.