Adaptive Parallel-World Self-Rewarding and Cross-Model Mutual Rewarding for Large Language Models

Abstract

Reward generation is a central component of reinforcement learning, preference optimization, and self-improvement in large language models (LLMs). Existing approaches have demonstrated that an LLM can evaluate its own outputs and use the resulting judgments to improve its behavior. However, self-rewarding systems may suffer from self-evaluation bias, limited evaluative diversity, and the accumulation of errors across iterative optimization. Cross-model evaluation can introduce additional perspectives, but relying exclusively on external evaluators may increase computational costs and create new dependencies. This paper proposes Adaptive Parallel-World Self-Rewarding and Cross-Model Mutual Rewarding (APSMR), a unified framework that dynamically coordinates self-rewarding and mutual rewarding across multiple LLMs and multiple evaluative branches of the same underlying model. In APSMR, a parallel world is defined as an independently configured evaluative branch derived from the same underlying model, with each branch assigned a potentially different reward orientation, evaluation rubric, reasoning strategy, or task perspective. The framework distinguishes intra-model parallel-world self-rewarding from inter-branch mutual rewarding and cross-model mutual rewarding. An adaptive controller selects among these modes according to estimated reward reliability, disagreement, uncertainty, task characteristics, computational cost, and observed feedback quality. A reliability-aware reward aggregation mechanism combines available judgments while accounting for evaluator dependence and calibration. The resulting rewards can support candidate selection, preference-pair construction, supervised refinement, or reinforcement-learning-based policy optimization. This paper formalizes the framework, presents its operational algorithm, proposes a multi-stage evaluation protocol, and identifies testable research hypotheses. The framework is intended to reduce dependence on a single evaluation pathway rather than assume that mutual rewarding is always superior to self-rewarding. Because APSMR is a proposed method, its comparative effectiveness remains an empirical question. The paper provides a reproducible experimental blueprint for evaluating reward accuracy, downstream task performance, robustness, calibration, computational efficiency, and resistance to reward manipulation.