End-to-end decision latency of a confidence-gated Laya–phi4 cascade on one consumer GPU

Abstract

A small, fast text classifier can answer the easy cases itself and pass only the uncertain ones to a large language model. Version 1.0 of this work estimated how much time such a two-stage setup saves; this version measures it while both models run on one consumer graphics card. The two-stage setup made the average decision take less than half as long as the large model alone. The version 1.0 formula for the expected decision time, fed with this run's separate timings of each model, matched the measured average closely. For the slowest items we could not tell whether the two-stage setup was faster or slower. The test items were repeated, so the large model's server sometimes found it had already processed the same prompt and could skip part of the work; a later estimate of the times without such whole-prompt reuse gives a larger average saving and still cannot tell for the slowest items. The large model was not tuned for speed, and this version does not show that the faster setup is as accurate. This version also corrects the size of the small model given in version 1.0: 322 million parameters, not 56 million. Changes in version 1.2. Version 1.2 changes text only. No number, measurement or analysis changed: the run, the frozen analysis and every figure are those of version 1.1 (DOI 10.5281/zenodo.23309069), and only the note file is renamed. Appendix A7 no longer offers the original data on request, an offer that contradicted withholding the item texts because two of the four source datasets state no licence; it now says plainly what a reader can and cannot check without them. Section 3.6 cites the prompt-cache literature for the mechanism behind the whole-prompt hits. Section 2 states that the 0.90 gate is a cut on the score the small model reports, not a demonstrated 90 % chance that an accepted label is right, and that the five-topic routing label set has no out-of-scope option; section 4 carries both as limitations. Appendix A1 supports the parameter-count correction with the publisher's model card. Appendix A11 lists each change and where it is. Details. A Laya text classifier (0.3.6 multilingual checkpoint) answers when its top probability is at least 0.90, otherwise the item goes to phi4:14b. Both models were resident on one RTX 4070, with phi4 served over localhost. The same 204-item holdout was run in six rounds of three arms (phi4 alone, Laya alone, cascade), 3672 timed decisions, under an internally preregistered, hash-frozen analysis plan. Each item's time in an arm is its median over the six rounds. Cascade minus phi4 alone, with 95 % bootstrap intervals over items: mean 100.7 against 226.8 ms, difference -126.0 ms [-142.1, -109.7]; median 17.0 against 223.7 ms, difference -206.7 ms [-246.3, -197.3]; 95th percentile 304.4 against 293.7 ms, difference +10.7 ms [-3.3, 31.5], undecided. The v1.0 cost model predicted the live mean (arithmetic mean over rounds, 101.1 ms) with an error of -1.0 ms [-2.3, 0.2]. Qualifications: the items recurred in every round and the phi4 server kept a prompt cache, so 29 % of phi4 calls in the phi4-alone arm and 39 % in the cascade found their whole prompt already processed; a post hoc estimate that adds back the prompt processing these hits skipped, keeping the reuse of each task's shared prompt prefix, gives a mean difference of about -145 ms and leaves the tail undecided. The phi4 baseline used the v1.0 settings, not tuned for speed. This run does not show that the cascade is as accurate as phi4 alone: on the first round both arms got 180 of 204 items right, though not the same items (six gains and six losses for the cascade). Package: the note (PDF; Markdown in the archive), the frozen plan and freeze file, run and analysis code with tests, per-call and per-block records without item texts, logs and reports. Item texts are not included. Code under MIT, everything else under CC BY 4.0 (file LICENSE); EXPORT_MANIFEST.json lists every file and every change made for publication.