FIE-sLLM Benchmark v1.1: ontology-constrained decoding for small open language models on clinical nutrition abstracts (code, results, raw outputs)

Abstract

Companion material for the article Ontology-Constrained Decoding for Small Open Language Models: Structural Gains, Content Trade-offs, and Silent Omission in Clinical Nutrition Abstract Extraction (submitted to IEEE Access, 2026). Six open small language models (Granite-4.0-1B, Qwen3-1.7B, Qwen3-4B-Instruct-2507, Phi-4-mini-instruct, Kanana-1.5-8B, Granite-4.1-8B) were run under four conditions (C0 free generation; C1 schema in prompt; C2 structure-constrained decoding; C3 ontology-constrained decoding with controlled vocabulary) and a rule baseline on 1,658 frozen PubMed abstracts on vitamin D and probiotics. Records were validated in two stages (JSON Schema, closed-world ontology rules) before knowledge-graph insertion and evaluated with gold-free structural metrics, agreement with 386 linked ClinicalTrials.gov registrations, and blinded two-judge human assessment of 360 records. The record contains three archives: (1) code, ontology v1.1 (OWL), extraction profile, controlled vocabulary, derived JSON Schemas, prompts, freeze manifests and corpus manifest (PMIDs and hashes; abstract text is not redistributed); (2) all result tables, paired statistics, per-run metrics, and the human-judgment analysis tables with reproducible code; (3) per-abstract raw model outputs and validation results for every run, KG exports, and ClinicalTrials.gov snapshots. See README.md inside each archive. Ontology v1.0 was first released in an earlier reproducibility package (10.5281/zenodo.20774211); this record supersedes it with ontology v1.1 (adds the hasArm property), a new frozen corpus (1,858 abstracts), and a different benchmark design. Code: MIT. Data and outputs: CC BY 4.0. Copyright holder: ANCHOR Program, Jeonju University; author: Dongwook Han. Funding: This research was supported by the Regional Innovation System & Education (RISE) program through the Jeonbuk State RISE Center, funded by the Ministry of Education (MOE) and Jeonbuk State, Republic of Korea (grant number: 2026-RISE-13-JJU). Version 1.1 (25 September 2026): revision after an independent review of the manuscript. The code and results archives are updated; the raw-outputs archive is unchanged from v1.0 (same file). Changes: (a) scripts/70_stats.py v2 and results/stats_eval.csv — for the registry metrics (primary-field agreement, valid-but-discordant rate) the 386 silver-standard abstracts are resampled by registered trial (318 NCT clusters) for the 95% bootstrap intervals, and the field-level McNemar test is replaced by a cluster sign-flip permutation test (10,000 permutations, seed 20260921) with Holm adjustment; the record-level McNemar test for KG insertion is unchanged; the metric formerly named valid_but_wrong is now valid_but_discordant. (b) New scripts/72_pruning_density.py with results/pruning_eval.csv (elements removed by the predefined normalization pruning rule, per model and condition) and results/density_common_eval.csv (nodes per record on abstracts inserted under both C1 and C3). (c) Figure 3 regenerated; figure/table build sources reduced to make_figs.py, build_tables.py and tables.json. (d) PubMed candidate lists (data/candidates/*.csv) added, documenting the query strings, search date (2026-06-16), relevance sort order and hit counts. No prompt, schema, normalization rule, corpus file or raw model output was changed. See "Changes in v1.1" in README.md.