Empirical Analysis of Small Language Model Quantization for CPU-Only Cloud Infrastructure

Abstract

As Large Language Models (LLMs) grow in complexity, the hardware required to serve them often exceeds the capabilities of standard cloud instances and edge devices. This article investigates the efficacy of Small Language Models (SLMs) specifically Llama 3.2 (1B) and Qwen 2.5 (1.5B) as cost-effective alternatives when paired with modern quantization techniques. By conducting empirical benchmarks on Google Colab’s CPU-only infrastructure, we evaluate the trade-offs between 4-bit (Q4_K_M) and 8-bit (Q8_0) integer quantization against a high-precision baseline. Our results demonstrate that 4-bit quantization on a standard CPU provides up to a 120% increase in inference throughput for 1.5B parameter models compared to 8-bit variants, while maintaining a negligible system RAM footprint (under 1GB). This study identifies the “sweet spot” for local and restricted-resource AI deployment, proving that aggressive quantization of SLMs enables real-time, scalable AI performance on general-purpose hardware without the necessity for dedicated GPU acceleration.