Selective Overconfidence: How Large Language Models Underserve Social Welfare Law Across Jurisdictions

Abstract

Large language models are increasingly used in legal information systems, yet their knowledge is not uniformly distributed across legal domains. We introduce a knowledge-probe pipeline that measures whether an LLM exhibits differential confidence across areas of law. Applying the pipeline to 1,200 court decisions across three jurisdictions (Germany, France, Canada), we find that Llama 3.3 70B systematically under-hedges on social welfare law relative to tax law in non-English jurisdictions: 49.8% of tax probes elicit admitted uncertainty vs. only 6.8% for social welfare in Germany (p < 0.0001), with a smaller but significant gap in France (p = 0.003) and no gap in Canada (English). The pattern is consistent with the representation of each language in training corpora and replicates on Qwen3 32B. The model achieves comparable accuracy in both domains, yet signals false confidence precisely in the legal areas that serve socioeconomically vulnerable populations. A retrieval-augmented intervention, providing related legal text at query time, closes 87% of the gap, suggesting the overconfidence stems from a knowledge deficit. We frame this as a form of compounding injustice: already marginalized users receive confidently stated but unreliable information in legal areas that are historically underserved and underrepresented in NLP, with no signal that the system's knowledge is thinner in their area of need than in other areas of law, which are much better hedged.