From Messy Country Names to ISO 3166 Codes: A Small Model That Abstains, and When a Lexicon Is Enough

Abstract

Company registers and forms record countries as free text: in any language, misspelled, abbreviated, historical, or not a place at all. We present CountryCode-Decide, a 74.7 MB open-weight model (mmBERT-small with a country head and a US-state head, 8-bit ONNX) that maps such a string, read with its field and register, to an ISO 3166-1 alpha-2 or ISO 3166-2:US code with a calibrated probability, and abstains with a short candidate list when a requested error rate cannot be met. Thresholds are certified by fixed-sequence testing on held-out real register strings. Generated training variants had to pass quality gates and a human audit of their labels; the first audit failed 9 of 10 releases, and we report the repair. On two business registers never used in development (New York and Oregon, 361 distinct strings, read once after a frozen pre-registration) the model is 88.6% accurate when it must answer every string, against 85.0% for a 35B-parameter LLM that is 29 times slower; all pre-registered hypotheses were met. An external review then showed two things we report here. First, an exact lexicon that applies the same context rules as the training labels reaches 89.5% on these registers, and on the 100 strings a person had to label the model, the LLM and that lexicon are level (61.0, 61.0 and 63.0%); the model's advantage appears on misspelled strings of the registers it was trained on, where it reaches 82.6% against 73.8% for the LLM and 35.8% for the lexicon. Second, the pre-registered 2% certificate counted the same string in several fields as independent draws; counted per register and normalised key, only the 5% target is certified, and the corrected release answers 87.8% of the fresh strings with a realised error of 2.5%, its errors concentrated on the hard strings. Code, weights and aggregate results are released under Apache-2.0.