Skip to content
Preprint

Vosti: Specifying, Implementing, and Verifying Deterministic LLM Inference

Sep 2026 · 2 citations · 36 references
Computer Science

TL;DR

Vosti, an inference engine designed and verified against this specification, produces bitwise-identical logits across every tested execution variation and achieves performance comparable to vLLM's batch-invariant mode on decode-heavy workloads, while providing a stronger, formally verified determinism guarantee.

Abstract

LLM inference systems may vary batch composition, prompt chunking, prefill/decode execution, and KV-cache reuse, eviction, or recomputation. These optimizations should not affect system outputs. Production systems, including vLLM's batch-invariant mode and SGLang's deterministic mode, target this goal but lack a formal system-level specification. We formalize deterministic LLM inference: under a fixed model and deployment configuration, requests with the same prompt and initial sampler state produce bitwise-identical logits at corresponding output positions across executions. Our tests find that these production modes produce different logits under some execution variations. To address this limitation, we present Vosti, an inference engine designed and verified against this specification. Vosti chooses kernels independently of runtime engine state and ties cached KV values to their logical token prefixes. Its proof decomposes at the engine/GPU kernel boundary: a Verus inductive proof establishes that scheduling and the paged, prefix-sharing KV-cache preserve output logits, while a Triton analyzer proves bitwise-equal selected kernel outputs across batches, query lengths, and paged KV-cache layouts. Vosti produces bitwise-identical logits across every tested execution variation and achieves performance comparable to vLLM's batch-invariant mode on decode-heavy workloads, while providing a stronger, formally verified determinism guarantee.

View source

Similar papers

Sep 2026

NebulaSD: Many-for-Many Speculative Decoding

NebastianSD is presented, a many-for-many, or M-for-N, speculative decoding system that organizes draft and target workers into independently schedulable resource pools and dynamically reconstructs stage-specific batches from shared request pools.

Jun-Hao He, Hong-Yang Du · 0 citations
Open access Oct 2026

Stage-Aware Optimization of Llama 3 Model Inference for Language Services on Many-Core CPUs

LLM-based language services (e.g., summarization, translation, and question answering) require low-latency and cost-effective inference, yet GPU resources may be costly, unavailable, or difficult to integrate in many deployments. Many-core CPUs provide a flexible alternative, but achieving efficient inference requires...

Lan-Xin Zhao, Dong Huang, Wan-Rong Gao et al. · 0 citations
#machine learning Preprint Oct 2026

APEX: Speculate smarter, not deeper

Speculative decoding reduces large language model inference latency by drafting multiple tokens before target-model verification, but its effectiveness depends on both the proposal mechanism and draft depth. Fixed configurations cannot respond to changes in predictability, repetition, and acceptance during generation,...

Manvi Jha, Zach Zhang, Zhi-Chao Xu et al. · 0 citations
Review Sep 2026

Herschel: Continuous Optimization of Production LLM Inference through On-Demand Profiling

Model-as-a-service platforms call for continuous optimization as complex serving conditions expose inefficiencies missed before deployment. Detailed always-on profiling can incur substantial overhead, while lightweight collection omits information needed for diagnosis. We present Herschel, a continuous optimization sys...

Lu-Ping Wang, Wei-Gao Chen, Yi-Fei Wu et al. · 0 citations
#artificial intelligence Preprint Sep 2026

vla.simd: Efficient CPU Inference for Language-Conditioned Manipulation

This work presents vla.simd, a CPU inference engine that combines shared SIMD micro-kernels, reusable computation, and target-specific optimization, and introduces IMPACT, an ACT-based policy with cached text representations and language-modulated visual features.

Khanh Duy Nguyen, Hoang M. Truong, A. T. Le · 0 citations
#artificial intelligence Preprint Sep 2026

Dynamic Adaptation of the LLM Context for Generating Routines with Coupled Semantics

LLM-based code generation fails when correctness depends on execution-dependent coupling: the meaning of one routine is defined by the runtime behavior of another, a relationship that cannot be resolved from textual descriptions alone. This limitation, which we call static binding, is not confined to explicitly coupled...

Gnaneswar Villuri, Hashmath Shaik, Alex Doboli · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.