Efficient and Interpretable Mixtures of Experts: Statistical Inference, Initialization, and Applications

Abstract

Scientific discovery increasingly relies on methods that are both flexible and interpretable. Traditional statistical models offer interpretability but depend on restrictive assumptions, whereas modern machine learning methods often sacrifice transparency for predictive accuracy. The Mixture-of-Experts (MoE) framework provides a principled compromise by combining multiple local regression models through a data-driven gating mechanism, capturing complex heterogeneous relationships while remaining interpretable. Despite this potential, practical MoE modeling is hindered by high computational cost, sensitivity to initialization, difficulty in selecting the number of experts, and limited tools for inference and implementation. We address these issues by developing a supervised-compression initializer that provides a data-driven initial guess for the number of clusters. We further propose likelihood-based procedures for confidence intervals and a fixed-gate bootstrap method for prediction intervals. Extensive simulations and case studies show that our MoE framework attains predictive performance comparable to or better than classical nonparametric and machine learning approaches, while providing clearer interpretability and more reliable extrapolation in data-scarce regions. Meanwhile, it achieves over an order-of-magnitude speedup compared with existing packages. These results demonstrate that MoE models can serve as a flexible, interpretable, and computationally efficient tool for modern data analysis.