When Logical Correctness Is Not Rational Judgment: The Limits of Formal Reasoning in Algorithmic Decision Systems

Abstract

Algorithmic failures are conventionally diagnosed as deviations from a correct specification — bias to be corrected, error to be patched. This article argues that the more consequential and less visible failure mode runs the other way: a system executes its specification exactly, and the specification was inadequate to the situation it governed. The distinction is developed through a formal argument rather than an analogy. Treating the feature set available to a learning system as a -algebra , the article shows that no -measurable function can represent a value-relevant event excluded from , and — more precisely — that no -measurable self-assessment functional can take the adequacy of  itself as an argument. Every reliability measure a model reports, from calibration to conformal coverage, is computed inside the representation whose adequacy is in question. Sections 2 and 3 locate this claim against the literatures on bounded rationality, rule-following, and algorithmic fairness, arguing that each stops short of the reflexive point at issue. Sections 4 through 6 develop the argument through decision theory, the Rashomon set, Goodhart-type selection effects, and an impossibility theorem for fairness criteria, and confront the two strongest objections available — that scope-monitoring is already being formalised, and that closure under a fixed representation cannot be a principled limit if human reasoners are themselves physical systems. The article concludes that formal validity and practical rationality answer to different criteria of success, that the empirical superiority of statistical prediction over human judgment does not collapse this distinction, and that the operative difference lies in the structure of failure rather than in its frequency: formal errors are quiet, mechanically correlated, and reproduced at scale, while human error — correlated though it often is by institutional bias, professional norms, and bureaucratic routine — remains comparatively noisy, locally visible, and costly to scale.