Multi-Objective Orchestration of Small Language Model Ensembles: Balancing Accuracy, Diversity, and Fairness

Abstract

Small Language Models (SLMs) offer efficient and practical alternatives to large-scale models in resource-constrained environments. We present a principled framework for constructing SLM ensembles that jointly optimize three competing objectives: prediction accuracy, output diversity, and fairness. Our method combines an interpretable cost-function formulation with a multi-objective evolutionary algorithm to discover Pareto-optimal ensemble configurations. We further introduce a two-stage combiner that produces diverse candidate responses and selects final outputs via embedding-based semantic consensus. Experiments on the MentalChat16k mental-health dialogue dataset show that the best-performing SLM ensemble configurations can match or surpass a fine-tuned Llama 3.1 70B model, achieving improvements of 0.86% in ROUGE-1, 5.84% in ROUGE-2, 4.93% in ROUGE-L, and 7.01% in semantic similarity. These results indicate that strategically orchestrated ensembles of small models can offer competitive or superior performance to significantly larger LLMs, while providing greater flexibility, interpretability, and accessibility for researchers operating under limited resources.