Prefill or prompt processing and Decode or token generation are two distinct subphases of LLM inference that are greatly influenced by LLM accelerators such as GPT-Generated Unified Format (GGUF Q4_K_M), NormalFloat 4-bit (NF4) Quantization, FlashAttention-2 and others. Although these accelerators clearly improve end-to-end LLM inference performance, their effectiveness over these subphases remains largely understudied. To address this gap, we present a cross-platform, multi-model empirical study, where we deploy multiple ∼ 1B-parameter LLMs on GPU, CPU, and Raspberry Pi 4B edge hardware platforms in the presence and absence of these accelerators. Each test case evaluates 10,000+ inference runs with separate phase-wise and end-to-end performance indicators. Our study brings several important observations, including the contrastive effect of quantization under different hardware bottlenecks, along with a quantification of runtime delays (up to +139%) caused by the lack of parallelism in the ARM architecture. Based on these benchmarking results and observations, we identify several open research challenges in the concluding section. Our work is fully reproducible and open-sourced on GitHub1.
Subhransu Das, Jiaming Cheng, Swathi Vallabhajosyula et al.· Practice and Experience in A...· 0 citations
Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation. What compresses well, however, need not deploy well. We survey dozens of recent works that report compression results on real hardware and extract practical deployment guidelines from them. Following these guidelines, we deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation. No single technique wins across tasks. For question answering, Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization, while structured pruning at the same precision costs 16 F1 at a 1% ratio. For segmentation, the ranking reverses: default quantization leaves parameters and MACs unchanged, whereas pruning cuts model size by nearly 80% at near-constant mIoU. Pruning can even inflate the deployed artifact by 21-49% by breaking k-quant super-block alignment; combined with longer, less format-compliant outputs, this raises Raspberry Pi latency up to 3.4x. Compression can also manufacture the appearance of competence rather than destroy it visibly: one LoRA-recovered variant stays fully parseable and holds 71% strict BoolQ accuracy while sending 97 of 100 predictions to a single class, at 52.6% balanced accuracy. We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition, and condense them into task-specific deployment research directions. The right technique depends on the task, the model, and the hardware. Our experiment code and artifacts are open-sourced at https://github.com/Arnavvvkumar/deployment
Subhransu Das, Jiaming Cheng, Arnav Kumar et al.· 0 citations