A novel RAIE taxonomy along four scaling dimensions is proposed, which optimizes the entire thought process through search algorithms and self-verification, and introduces a task-oriented guideline for choosing the best TTS strategy.
Improving the task performance of Large Language Models (LLMs) is essential, yet scaling these models faces significant challenges such as diminishing returns and high costs. Multi-Agent Systems (MAS) offer a promising solution by distributing tasks among specialized agents to improve the overall task performance. This can reduce training costs at the expense of increased test time due to the discussion and decision-making process. The decision protocol is a critical component of MAS because it specifies how multiple agents collaborate to create a final solution. This thesis introduces the Multi-Agent LLM (MALLM) framework, which implements and evaluates various decision protocols, namely voting, consensus, and judge decision mechanisms, to simulate multi-agent discussions for conversational task solving. Unlike previous work that used a single decision protocol or tested them on limited datasets, this study systematically examines their impact on a diverse set of tasks, ranging from knowledge-based datasets (MMLU, MMLU-Pro, GPQA) and logic-based datasets (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0). The results indicate that consensus protocols excel in knowledge-intensive domains while voting and judge protocols are more effective for logic-based tasks. Increasing response diversity through independent solution generation improves decision quality, while changes in information access during the decision process have minimal impact.
: Large Language Models (LLMs) currently lack the robust command and decision-making (C&D) capabilities essential for the command and control domain. To address this critical gap, this paper proposes an emergence mechanism that integrates a domain-specialized Chain of Thought (CoT) framework with a Process Reward Model (PRM)-inspired evaluation and inference-time optimization paradigm. We construct a novel Chain of Command and Decision (CoCD) framework, a C2-specific CoT structure with contextual persistence, knowledge accumulation, and a human-in-the-loop feedback loop, and define a four-dimensional PRM-inspired evaluation framework for process-level assessment of C&D reasoning. Experimental evaluations on 40 C&D scenarios of varying complexity demonstrate that the CoCD framework significantly outperforms direct prompting (Mann–Whitney U = 1314, p < 0.0001, Cohen’s d = 1.340) and Standard-CoT ( p = 0.005, d = 0.606) in composite performance. PRM-guided Best-of-N selection further improves performance by 5.8% over single-sample CoCD ( p < 0.001, d = 0.855), providing direct empirical evidence for the utility of process-aware reward signals at inference time. CoCD’s structural advantage is greatest in high-uncertainty, structurally ambiguous scenarios (Level 3 gap: + 0.925 points), revealing a complexity-type effect that informs the deployment scope of structured CoT frameworks. These findings provide empirical support for domain-specialized structured reasoning and process-level evaluation as foundations for future RL-based C&D capability development in LLMs.
Yazhi Zheng, Xiao-Long Cui, Xin Wang et al.· Computers, Materials & C...· 0 citations
Long-horizon tasks require sustained perception, reasoning, and exploration, and are a persistent challenge for large language model (LLM) agents. This gap is reflected in their limited performance on continual learning benchmarks such as ARC-AGI-3, especially when models are evaluated out of the box. Various agent harnesses have been proposed to close this gap, and each commits to a strategy for handling long sequences of observations, i.e., what information to save from the environment and how to load it into model context, a choice we argue is particularly consequential. Existing methods for context management face a significant tradeoff, as preserving more information makes retrieving relevant details less tractable. We propose PRO-LONG, a minimal context management framework built around programmatic memory for LLM agents in long-horizon, exploratory settings. PRO-LONG addresses the tradeoff by keeping a complete, structured interaction log and capitalizing on recent progress in coding agents to search this history efficiently. On the full ARC-AGI-3 public game set, PRO-LONG improves over a base coding agent by an average of 18.0 percentage points across frontier models, and matches or exceeds state-of-the-art specialized harnesses (up to 76.1% pass@1) while using 4.2-5.8x fewer tokens. With Fable 5, PRO-LONG achieves 97.4% best@2 at a total cost of \$1,750. Relevant code and logs are available at https://github.com/alexisfox7/PRO-LONG.
A. Fox, Junlin Wang, P. Rosu et al.· 2 citations· ⚡1
PoTRE (Poly-Topological Reasoning Ensembles), a heterogeneous framework that decouples inference into four agents that achieves improved reasoning performance using similar or fewer inference tokens compared to heavily scaled homogeneous baselines is introduced.
This project explores the Countdown arithmetic reasoning task: given a set of numbers, produce an arithmetic expression that evaluates to a target value on the Qwen 2.5-0.5B base model and proposes two complementary extensions targeting these failure modes.
This paper investigates the performance of Large Language Models (LLMs) as autonomous agents in the Hearthstone digital collectible card game. Although traditional numerical agents have demonstrated strong results in competitive game environments, the reasoning capabilities of LLM-based agents remain largely unexplored in this context. To address this gap, we developed an LLM-driven Hearthstone agent using the Sabberstone framework to evaluate several models, including GPT-4o, GPT-4o-mini, o3-mini, and GPT-5-mini, across multiple decks and prompting strategies. Our experiments compare their win rates against established numerical agents and analyze the impact of different prompting techniques, such as Chain-of-Thought (CoT), Reverse Chain-of-Thought (RCoT), ReAct, and Directed Acyclic Graph (DAG) prompting. The results indicate that all evaluated LLMs outperform the random baseline, and GPT-5-mini achieves win rates close to the strongest numerical agents under our evaluation setting. Among the prompting strategies, DAG achieved the highest win rate, while ReAct performed significantly worse. These findings highlight both the current limitations and the promising potential of LLMs for strategic reasoning and decision-making in complex, rule-based digital games such as Hearthstone. Leveraging their natural language understanding capabilities, LLMs have the potential to act as intelligent virtual companions, guiding and supporting players through the initial phases of gameplay.
Christian Poglitsch, Philipp Bardakji, Johanna Pirker· International Conference on...· 0 citations