This paper investigates a hybrid reinforcement learning setting in tabular Markov Decision Processes (MDPs), where an agent aims to learn an optimal policy by combining online interactions with a target environment and offline data from a source environment. A central challenge is that offline data may be collected from outdated environments with shifted transition dynamics, making naive integration of historical data ineffective. To address this, we propose a unified algorithmic framework featuring two algorithms: MIN-UCB-VI for regret minimization and MAX-LCB-VI for best policy identification. Both algorithms leverage fine-grained bias information to more effectively exploit offline data under general transition shifts. We provide theoretical guarantees for our framework, including both instance-dependent and independent upper bounds on regret and sub-optimality gap. Furthermore, we establish matching lower bounds to demonstrate the optimality of our approach and validate our theoretical findings through extensive experiments.
With the rapid advancement of large language models (LLMs), multi-agent systems have emerged as a promising alternative to scaling up a single model. Existing approaches ensemble multiple LLMs to improve response quality, but they often rely on static prior knowledge of model capabilities and prompts, and require extensive parameter tuning. Some of the methods also treat each combination of LLMs as a learning objective, which leads to exponential time complexity. In this work, we propose an offline-to-online combinatorial experts selection (CES) framework to address these limitations. CES leverages offline evaluation to warm-start model capability estimation and employs online learning to adapt to capability shifts and correct offline inaccuracies. By integrating model features and input semantic representations into a combinatorial multi-armed bandit formulation, CES captures the interaction between prompts and LLMs without introducing complex auxiliary structures such as knowledge graphs. Modeling each LLM as a base arm with answer quality represented by a linear function of model and prompt features, CES achieves polynomial time complexity while excellently balancing performance and cost. Our experiments, conducted on popular LLM evaluation datasets such as AlpacaEval 2.0, show CES's effectiveness, laying the groundwork for future extensions.
Jinkun Xu, Minghan Wang, Zhiyong Wang et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
