Mathematical Methods and Algorithms for Large Language Model Token Optimization in Modern Agentic Artificial Intelligence Enterprise Systems

Abstract

Large language models now sit at the centre of enterprise agentic pipelines that plan, retrieve, call tools, and generate answers under strict cost and latency budgets. In these systems the token is the fundamental unit of both computation and price, so the mathematics of spending tokens efficiently has become a first order engineering concern. This paper presents a unified mathematical and algorithmic account of token optimization across four levels of the stack: subword tokenization, prompt compression, key value cache management, and agent loop context governance. We formalize each level as an optimization problem, namely minimum expected sequence length for tokenization, budgeted information maximization for prompt compression, and submodular cache eviction for inference memory. We implement byte pair encoding from first principles and report reproducible measurements of fertility and compression on four enterprise content regimes, observing token fertilities between 1.26 and 2.01 and effective compression between 4.29 and 8.59 characters per token at a vocabulary of 8096 units. An information theoretic pruning analysis shows that ranking tokens by self information retains content faster than uniform removal, and a transparent parametric cost model shows that compressing retrieved context by sixty five percent lowers per episode token consumption by approximately fifty four percent for a six round agent. The results give practitioners a rigorous basis for controlling the cost of agentic deployments without sacrificing task fidelity.