Beyond Generation: What a Decision-Native Model Does Well in a Long Task, and What It Needs from Its Harness

Abstract

Many automated workflows end in a bounded decision among options that the software already knows, yet such decisions are usually obtained from language models by generating text. Decision-native models instead return one of the supplied options together with a probability distribution. We study one of them, Jev, on the Three-Phase Vault, a puzzle small enough to be solved exactly (65,375 states, a 26-move optimum), so that every step of every run can be scored against the optimum. On 400 fixed states with identical inputs, Jev chose an optimal action more often than three reasoning models (53.0% against 28.5–36.3%), at a median latency of 0.41 seconds against 2.3–4.4 seconds. Over whole runs, however, Jev often undid its own progress and completed 22% of runs under a minimal harness, against 86%, 32% and 26% for the reasoning models. A harness that kept a persistent goal and filtered the options by it raised Jev's completion from 0% to 94% (141 of 150 runs), and replaying the original code three and a half weeks later gave 48 of 50. Small changes then reduced completion sharply: shuffling the order of the options (18 of 50), rewording the goal instruction (32 of 50) and keeping drop actions in the filtered set (4 of 70). On twelve generated environments the filter's gain was not statistically reliable, and removing drop actions, which helped on the Vault, lowered completion there from 73% to 13%. The performance of a decision-native model on a long task thus depends on a harness whose effect is large and fragile, and building harnesses that transfer is the main open problem. Preprint, not peer reviewed. Data, code and all recorded runs: doi:10.5281/zenodo.23323117.