End-to-end decision latency of a confidence-gated Laya–phi4 cascade on one consumer GPU

Abstract

A small, fast text classifier can answer the easy cases itself and pass only the uncertain ones to a large language model. Version 1.0 of this work estimated how much time such a two-stage setup saves; this version measures it while both models run on one consumer graphics card. The two-stage setup made the average decision take less than half as long as the large model alone. The version 1.0 formula for the expected decision time, fed with this run's separate timings of each model, matched the measured average closely. For the slowest items we could not tell whether the two-stage setup was faster or slower. The test items were repeated, so the large model's server sometimes found it had already processed the same prompt and could skip part of the work; a later estimate of the times without such whole-prompt reuse gives a larger average saving and still cannot tell for the slowest items. The large model was not tuned for speed, and this version does not show that the faster setup is as accurate. This version also corrects the size of the small model given in version 1.0: 322 million parameters, not 56 million. Details. A Laya text classifier (0.3.6 multilingual checkpoint) answers when its top probability is at least 0.90, otherwise the item goes to phi4:14b. Both models were resident on one RTX 4070, with phi4 served over localhost. The same 204-item holdout was run in six rounds of three arms (phi4 alone, Laya alone, cascade), 3672 timed decisions, under an internally preregistered, hash-frozen analysis plan. Each item's time in an arm is its median over the six rounds. Cascade minus phi4 alone, with 95 % bootstrap intervals over items: mean 100.7 against 226.8 ms, difference -126.0 ms [-142.1, -109.7]; median 17.0 against 223.7 ms, difference -206.7 ms [-246.3, -197.3]; 95th percentile 304.4 against 293.7 ms, difference +10.7 ms [-3.3, 31.5], undecided. The v1.0 cost model predicted the live mean (arithmetic mean over rounds, 101.1 ms) with an error of -1.0 ms [-2.3, 0.2]. Qualifications: the items recurred in every round and the phi4 server kept a prompt cache, so 29 % of phi4 calls in the phi4-alone arm and 39 % in the cascade found their whole prompt already processed; a post hoc estimate that adds back the prompt processing these hits skipped, keeping the reuse of each task's shared prompt prefix, gives a mean difference of about -145 ms and leaves the tail undecided. The phi4 baseline used the v1.0 settings, not tuned for speed. This run does not show that the cascade is as accurate as phi4 alone: on the first round both arms got 180 of 204 items right, though not the same items (six gains and six losses for the cascade). Package: the note (PDF; Markdown in the archive), the frozen plan and freeze file, run and analysis code with tests, per-call and per-block records without item texts, logs and reports. Item texts are not included. Code under MIT, everything else under CC BY 4.0 (file LICENSE); EXPORT_MANIFEST.json lists every file and every change made for publication. Note added 12 October 2026: the latest version of this record is version 1.2 (DOI 10.5281/zenodo.23315083), which corrects and extends the text of this version and does not change any numerical result reported here. Appendix A7 drops the offer of data on request and names instead the public files and the join that reproduce the accuracy counts; three references were added (Gim et al. 2024; Geifman and El-Yaniv 2017; Guo et al. 2017); the 0.90 gate is stated as a cut on the small model's own confidence score, not a guarantee of 90 % correctness; the routing label set is stated to have no out-of-scope category; the correction of the small model's size to 322 million parameters now cites the publisher's model card; appendix A8 records the model reviews; and a new appendix A11 lists each change. No measurement or analysis was repeated, and the deposit of version 1.2 differs from this one only in the note file, the export script and EXPORT_MANIFEST.json. The files of this version remain unchanged and the record remains available under the same DOI. All versions: DOI 10.5281/zenodo.22922768.