Neuro-Elastic: A Unified Framework for Hardware-Aware Adaptive Quantization and Dynamic Sparsity in Real-Time Edge Intent Prediction

Abstract

Deploying compact Transformer encoders on resource-constrained edge devices forces a conflict between reasoning quality and tight latency, thermal, and energy budgets, yet static compression ignores both the variance in input difficulty and the changing state of the hardware. We propose Neuro-Elastic, an adaptive inference framework that operates at two granularities: per-input sparsity (entropy-driven early exit and token pruning) and device-state-driven switching among pre-compiled mixed-precision model variants. Its central contribution is this hardware and software co-design, not the individual mechanisms, which are established. Three components realize it: Hardware-Aware Differentiable Mixed-Precision Quantization (HA-DMPQ), which learns layer-wise bit-widths by gradient descent within the target accelerator’s native instruction set; LoRAQuant, a split-precision low-rank strategy for on-device fine-tuning; and Dynamic Sparsity Induction (DSI), which cuts memory bandwidth through entropy-driven exits and cache-aware token pruning. We measure all latency and energy on an NVIDIA Jetson Orin Nano and use a Qualcomm Hexagon 680 simulator only for instruction-level validation, scoping our claims accordingly. On the MASSIVE intent-classification benchmark with an mBERT-base backbone, the Eco operating point cuts latency by 46.2% and energy by 54.5% at a 1.6 percentage-point accuracy cost (95.2% versus the 96.8% FP16 teacher), while the Balanced operating point stays within 0.3 points of the teacher (96.5%) and still cuts latency by 31.9%. All results are reported as mean ± standard deviation over five independent runs.