Mind the Generalization Gap: Lessons from Reproducing Research on Machine Learning for Wireless Networks

Abstract

Machine learning is increasingly used across wireless systems, with research papers reporting highly promising results, yet most proposed solutions are never deployed in practice. One major reason is the generalization gap between research evaluation and real world wireless settings. In this paper, we reproduce studies that apply machine learning to wireless systems across three application areas: throughput prediction, channel estimation, and sensing from wireless signals. We examine how data collection, processing, and splitting decisions affect reported performance. Across these case studies, we find that common evaluation practices, such as random assignment of samples to training and test sets, can introduce relationships between training and test sets that would not exist in the proposed deployment setting. These arise from temporal, environmental, or subject correlations in wireless measurements and can lead to overly optimistic estimates of model performance. When we redesign the evaluation to better reflect intended deployment settings, the estimated model performance is much worse, revealing substantial generalization gaps. We conclude with practical recommendations for designing and reporting machine learning evaluations for wireless systems, emphasizing data partitioning strategies that reflect the intended deployment setting.