When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning

Abstract

Legal reasoning requires identifying which version of a statute applies to a case, typically based on when the legally relevant events occurred rather than when the case is adjudicated. Yet whether large language models (LLMs) can reliably perform such temporal legal reasoning remains largely unexplored. We formalize this task as Temporal Applicable-Law Determination (TALD), and construct a TALD-Bench, a benchmark of 51,956 Chinese civil judgments spanning the enactment of the Civil Code. Evaluating nine advanced LLMs, we identify three key findings. First, models exhibit a strong bias toward newer law, achieving over 70% accuracy on new-law cases but below 25% on old-law cases. Second, models demonstrate substantial knowledge of statutory versions and temporal-effect rules, yet struggle to apply this knowledge to cases. Stronger general reasoning performance does not consistently translate into better TALD accuracy, whereas temporal-law hints yield substantial improvements, suggesting a bottleneck in how models deploy their existing capabilities. Third, temporal-law hints also increase uncertainty during statutory-version reasoning, suggesting that they may help models reconsider their default preference for newer law. Our findings highlight the importance of adapting how LLMs apply existing capabilities, rather than relying solely on improvements in legal knowledge or general reasoning.