Toolkit for Confidence-Corpus Consistency, Corpus Absorption and Rule Learning via Fine-Tuning on a Fabricated Corpus

Abstract

An open toolkit that fine-tunes a small causal language model on a corpus asserting one fabricated arithmetic answer for each of the 81 single-digit addition pairs, and compares the model's post-fine-tuning confidence in each fabricated answer against its own pre-fine-tuning confidence in the corresponding true answer, using an unchanged measurement procedure throughout. The toolkit implements fact-space generation, token-length-aware confidence measurement, baseline validation, fabricated-corpus construction, fine-tuning, post-fine-tuning re-measurement, and paired before/after visualization, and is documented as a reproducible instrument for probing whether a model's confidence tracks consistency with its own training corpus rather than correspondence with fact. Version 2 adds recovery after fabricated training, simple and conditional rule learning with controls, held-out additions, exact confidence intervals, and complete saving of figures and tables.