FIE-sLLM Benchmark v1.2: ontology-constrained decoding for small open language models on clinical nutrition abstracts (code, results, raw outputs)

Abstract

Companion material for the article Ontology-Constrained Decoding for Small Open Language Models: Structural Gains, Content Trade-offs, and Silent Omission in Clinical Nutrition Abstract Extraction (submitted to IEEE Access, 2026). Six open small language models (Granite-4.0-1B, Qwen3-1.7B, Qwen3-4B-Instruct-2507, Phi-4-mini-instruct, Kanana-1.5-8B, Granite-4.1-8B) were run under four conditions (C0 free generation; C1 schema in prompt; C2 structure-constrained decoding; C3 ontology-constrained decoding with controlled vocabulary) and a rule baseline on 1,658 frozen PubMed abstracts on vitamin D and probiotics. Records were validated in two stages (JSON Schema, closed-world ontology rules) before knowledge-graph insertion and evaluated with gold-free structural metrics, agreement with 386 linked ClinicalTrials.gov registrations, and blinded two-judge human assessment of 360 records. The record contains three archives: (1) code, ontology v1.1 (OWL), extraction profile, controlled vocabulary, derived JSON Schemas, prompts, freeze manifests and corpus manifest (PMIDs and hashes; abstract text is not redistributed); (2) all result tables, paired statistics, per-run metrics, and the human-judgment analysis tables with reproducible code; (3) per-abstract raw model outputs and validation results for every run, KG exports, and ClinicalTrials.gov snapshots. See README.md inside each archive. Ontology v1.0 was first released in an earlier reproducibility package (10.5281/zenodo.20774211); this record supersedes it with ontology v1.1 (adds the hasArm property), a new frozen corpus (1,858 abstracts), and a different benchmark design. Code: MIT. Data and outputs: CC BY 4.0. Copyright holder: ANCHOR Program, Jeonju University; author: Dongwook Han. Funding: This research was supported by the Regional Innovation System & Education (RISE) program through the Jeonbuk State RISE Center, funded by the Ministry of Education (MOE) and Jeonbuk State, Republic of Korea (grant number: 2026-RISE-13-JJU). Version 1.1 (25 September 2026): revision after an independent review of the manuscript. The code and results archives are updated; the raw-outputs archive is unchanged from v1.0 (same file). Changes: (a) scripts/70_stats.py v2 and results/stats_eval.csv — for the registry metrics (primary-field agreement, valid-but-discordant rate) the 386 silver-standard abstracts are resampled by registered trial (318 NCT clusters) for the 95% bootstrap intervals, and the field-level McNemar test is replaced by a cluster sign-flip permutation test (10,000 permutations, seed 20260921) with Holm adjustment; the record-level McNemar test for KG insertion is unchanged; the metric formerly named valid_but_wrong is now valid_but_discordant. (b) New scripts/72_pruning_density.py with results/pruning_eval.csv (elements removed by the predefined normalization pruning rule, per model and condition) and results/density_common_eval.csv (nodes per record on abstracts inserted under both C1 and C3). (c) Figure 3 regenerated; figure/table build sources reduced to make_figs.py, build_tables.py and tables.json. (d) PubMed candidate lists (data/candidates/*.csv) added, documenting the query strings, search date (2026-06-16), relevance sort order and hit counts. No prompt, schema, normalization rule, corpus file or raw model output was changed. See "Changes in v1.1" in README.md. Version 1.2 (12 October 2026): revision after a second independent review of the manuscript. The code and results archives are updated; the raw-outputs archive is unchanged from v1.0 (same file). Changes: (a) new scripts/74_review2_checks.py with results/review2_checks.json — normalization-pruning × consensus-omission cross-tabulation on the human-judged records; decomposition of the human-judged valid-but-wrong rate by insertion status (inserted under both C1 and C3 / C3 only / C1 only); sensitivity analysis of the KG-insertion contrasts with abstracts sharing a registered trial grouped as clusters; registry metrics excluding the 19 evaluation abstracts whose trials also have a development-set abstract. (b) results/secondary_registry_eval.csv (duration agreement vs. the registered primary-outcome time frame; primary-outcome recall) and results/token_decomp_eval.csv (whitespace share and compact-serialized length of raw outputs, C1 vs. C3). (c) results/stats_eval.csv written with six significant digits (values unchanged). (d) README documents implementation facts confirmed by code inspection: C1–C3 prompts are byte-identical; the lenient parser is used only for C0; disable_any_whitespace applies only to C2 and C3; stage-2 checks positivity only; the decoding grammars do not enforce minItems or the numeric lower bound. No prompt, schema, normalization rule, corpus file or raw model output was changed. See "Changes in v1.2" in README.md.