Is this Citation on Point?

Abstract

In 2023, a New York judge sanctioned two attorneys in \emph{Mata v. Avianca} for filing a brief containing hallucinated citations generated by ChatGPT. These failures can be largely caught by database lookups. The harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered, a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora. We either replace the cited case or change only the pinpoint page within the same case. Our evaluation of fourteen model configurations on the resulting examples shows that models catch $93$--$100\%$ of wrong-case corruptions, but they only catch $37$--$61\%$ of wrong-pinpoint corruptions on court opinions and $52$--$83\%$ on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Larger-scale models and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses $40\%$ of pinpoint mismatches on court opinions and $18\%$ on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. These results show that citation misrepresentations remain difficult to detect automatically. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.