Deploying Language Models on Android-Based Edge Devices: A Practical Evaluation Pipeline
TL;DR
Results show that most models above 500 million parameters were not suitable for the target environment, while a subset of 4-bit quantized models achieved stable execution and acceptable response quality, and provide practical guidance for integrating generative AI into low-resource consumer hardware.
Abstract
This paper presents a practical evaluation pipeline for deploying small language models (SLMs), compact versions of large language models designed to run with reduced memory and computational requirements, on constrained Android-based edge devices, using Android TV as a representative case. The study investigates both deployment feasibility and software-level acceleration strategies, such as the choice of the inference engine responsible for executing the model on the device, under strict memory and processing limitations. Results show that most models above 500 million parameters were not suitable for the target environment, while a subset of 4-bit quantized models achieved stable execution and acceptable response quality. In addition, the experiments demonstrate that the choice of inference engine has a major impact on performance, with MNN significantly outperforming llama.cpp on ARM-based devices (the processor family that dominates mobile and embedded hardware). These findings provide practical guidance for integrating generative AI into low-resource consumer hardware.