Observational Equivalence of LLM and Human Annotation
It is argued that the central challenge of text annotation is no longer choosing between human and machine coders, but developing coding rules that minimize ambiguity and accounting for the ambiguity that remains, and it is proposed using disagreement across LLMs to identify difficult cases and refine codebooks.
Kentaro Nakamura, Jing-Ling Tan, George Yean
· 0 citations