Ptarmigan: Zero-Code GPU Tensor Virtualization for Memory-Constrained LLM Inference

Abstract

Serving large language models (LLMs) on commodity edge and enterprise accelerators is fundamentally constrained by high-bandwidth device memory capacity. Existing parameter offloading solutions either mandate intrusive model code refactoring and custom computational kernel integration, or suffer from severe pipeline stalls and internal memory fragmentation due to dynamic allocation behaviors in framework runtimes. We present Ptarmigan, a non-invasive tensor virtualization runtime that achieves transparent layer-wise offloading with zero model modification. Ptarmigan guarantees deterministic memory bounds by pre-allocating an isolated two-slot physical buffer on the device and continuously pipelining flattened parameter blocks using an event-driven DMA scheduler. Dynamic layer pointer rebinding via standard PyTorch forward hooks enables transparent execution while entirely eliminating framework-level memory fragmentation. On an NVIDIA Tesla T4 GPU running TinyLlama-1.1B under identical 500 MB memory budgets, Ptarmigan delivers a 2.22x decoding throughput speedup (3.67 vs. 1.65 tokens/s) over standard Hugging Face offloading, capping peak parameter allocation at 602.25 MB versus 871.28 MB in the baseline. Under prompt scaling, compute latency progressively hides asynchronous transfer latency, delivering up to a 4.72x speedup during prefill execution while maintaining flat latency (+3.6% from 64 to 512 tokens). Micro-architectural ablations confirm that pinned flat-buffer packing reduces per-layer transfer overhead by 90% compared to pageable host memory.