Diagnosing Chinese Legal LLMs Across Memorization, Understanding, and Application: The Newtouch-LawBench Benchmark

Abstract

With the emergence of large language models (LLMs) as strong performers across diverse natural language tasks, high-stakes and knowledge-intensive domains such as law impose stricter requirements on factual reliability, norm-conformant application, and verifiable reasoning. Although existing Chinese legal evaluation benchmarks cover a range of tasks, they often suffer from flattened evaluation dimensions, limited attribution-oriented diagnostics, and overly singular metrics for generative tasks, making it difficult to precisely characterize model capability gaps across distinct cognitive stages of “memorization–understanding–application.” To address these limitations, we propose Newtouch-LawBench, a hierarchical evaluation framework tailored to the Chinese statutory law system. Drawing on Bloom’s taxonomy and the cognitive assessment characteristics of China’s National Legal Professional Qualification Examination, we reorganize legal competence into three progressive levels: legal knowledge memorization (L1), legal knowledge understanding (L2), and legal knowledge application (L3). Building on this framework, we constructed an evaluation dataset comprising 10 core subtasks with 7673 high-quality annotated instances, covering task paradigms including generation, extraction, and single-label selection. Based on a systematic evaluation of 14 representative LLMs, we find that: 1) performance is strongly task-dependent rather than monotonically ordered by cognitive level: legal element extraction is comparatively mature, whereas statutory recitation, temporal-anchor legal facts, disputed-issue identification, and criminal-charge prediction remain challenging; 2) legal fact verification in consultation settings and legal document generation exhibit substantial cross-model dispersion under ECLQS; 3) tasks involving Temporal-Anchor Legal Facts show a cliff-like performance distribution; and 4) the task-balanced Macro Overall score provides a compact diagnostic summary but should not be interpreted as a definitive model ranking. These findings provide empirical evidence for diagnosing Chinese legal LLM capabilities and for guiding optimization directions, such as evidence/statute alignment, retrieval augmentation, and structured modeling of long-form legal documents. The benchmark is available at https://github.com/newtouch-cloud/newtouch-lawbench