This research bridges theoretical foundations of reinforcement learning and graph-based memory with autonomous agent workflows, and offers a practical, scalable reference framework for developing artificial intelligence technologies in complex, multi-step autonomous systems.
Abstract
This paper addresses key technical challenges in current large language model (LLM) agent applications, including long-horizon planning, sparse reward attribution, and dynamic environmental interaction, by designing and optimizing an intelligent agent workflow. The proposed architecture is based on the synthesis of core AI paradigms: Visual, Language, Generative, Graph, Multimodal, Reinforcement, and Agent Intelligence. Unlike conventional baseline models that rely on static prompting and lack robust perception-action loops, our approach introduces a Partially Observable Markov Decision Process (POMDP) routing mechanism. This mechanism is augmented with an internal, self-correcting reward model that evaluates decision trajectories before execution. By integrating multimodal inputs and advanced reinforcement learning principles (such as proximal policy optimization and value function approximation), the agent maintains long-term structural memory and dynamically adapts its reasoning pathways to mitigate error accumulation. Empirical experiments on the ALFWorld embodied simulation environment and the WebShop online navigation benchmark demonstrate a 24.5% absolute improvement in task success rate and trajectory efficiency over mainstream baselines like the standard ReAct framework. Comprehensive ablation studies confirm the significant contribution of the reward-driven critique module in suppressing hallucination rates. This research bridges theoretical foundations of reinforcement learning and graph-based memory with autonomous agent workflows. Ultimately, the resulting architecture offers a practical, scalable reference framework for developing artificial intelligence technologies in complex, multi-step autonomous systems. Code is available at https://github.com/01Amez/RLAW_Implementation.
: Multi-Agent systems powered by Large Language Models have demonstrated remarkable capabilities; however, their effectiveness is limited by rigid, manually designed workflows that do not adapt to varying task complexity. Existing adaptive methods utilize greedy policies that select workflow components without antic-ipating downstream cost-accuracy tradeoffs, often necessitating multiple attempts to identify successful configurations. This work introduces AgentSearch, a cost-aware Monte Carlo Tree Search (MCTS) framework that constructs agentic workflows through deliberative lookahead search. The proposed approach employs a dual-network architecture: a value network that decomposes expected rewards into success probability and remaining cost estimates, and a grammar-constrained policy network that ensures semantically valid constructions. Guided by these networks, MCTS explicitly simulates candidate workflow trajectories, enabling principled reasoning about the cost-accuracy tradeoff prior to decision-making. The networks are trained using a multi-phase protocol that combines stochastic exploration, supervised pre-training, and self-play refinement. Experiments on mathematical reasoning (MATH, AQUA-RAT) and code generation (HumanEval, MBPP) benchmarks demonstrate that the method achieves 80.03% average accuracy, surpassing Chain-of-Thought by 13.78 points and the best multi-agent baseline by 8.87 points. Notably, the approach attains single-episode success while reducing computational costs by up to 47%, thereby eliminating the trial-and-error exploration required by previous adaptive methods.
D. Attota, Ying Xie· Proceedings of the 15th Inte...· 0 citations
Large Language Models (LLMs) have recently shown strong capabilities in reasoning, planning, and tool-use, enabling new forms of autonomous agents. However, LLM-based agents struggle with long-horizon sequential decision tasks that require precise action optimization and environment interaction. Reinforcement Learning (RL), while effective for sequential control, often lacks the high-level abstraction and task decomposition abilities needed for complex scenarios. This paper introduces an LLM-Augmented Reinforcement Learning Agent that integrates LLM-driven planning with RL-based action optimization. The proposed architecture leverages the LLM to generate subgoals, structured plans, and contextual guidance, while the RL agent refines low-level actions through interaction with the environment. Experiments on sequential decision tasks demonstrate improved sample efficiency, higher success rates, and more coherent action trajectories compared to RL-only and LLM-only baselines. This hybrid paradigm highlights a promising direction for building more capable autonomous systems.
Christophe D. Hounwanou, John Emeka Eze, Yaé Ulrich Gaba· 0 citations
Embodied planning requires agents to make coherent multi-step decisions based on dynamic visual observations and verbal goals. While recent vision-language models (VLMs) excel at static perception tasks, they struggle in interactive environments. Reinforcement learning (RL) offers a natural way to address this limitation, yet online RL approaches suffer from costly interaction and sparse rewards in embodied settings. This paper introduces ORBIT , an O n-policy R einforcement fine-tuning (RFT) framework with offline rewards for Em B od I ed T ask Planning, that preserves the generalization benefits of RFT while addressing the challenges of costly interaction and sparse rewards, supported by solid theoretical guarantees. Our approach is evaluated on EmbodiedBench, a recent benchmark for interactive embodied tasks, covering both in-domain and out-of-domain scenarios. Experimental results show that ORBIT achieves SOTA performance on EB-ALFRED, outper-forming all closed-source and online-RL-based methods, while being substantially more effi-cient in training speed and computational cost, remaining robust to sub-optimal expert trajectories, and exhibiting strong generalization to unseen environments. We released all code and data at https://github.com/mail-taii/Reinforced-Reasoning-for-Embodied-Planning
Di Wu, Jiaxin Fan, Chloe Gu et al.· Annual Meeting of the Associ...· 0 citations
This work proposes a reinforcement learning-based framework for fine-tuning and optimizing Small Language Models (SLMs) used in Agentic AI systems, with a focus on improving tool invocation and decision-making over complex, multi-step tasks. The framework targets both open-source and proprietary language models operating in interactive environments that integrate multiple data sources and external APIs through autonomous agents. We leverage Grouped Relative Policy Optimization (GRPO) to generate and evaluate multiple complete execution trajectories per query within a simulated interaction environment, assigning trajectory-level rewards through an autonomous LLM-based evaluation module (RULER). High-reward trajectories are used to iteratively update SLM parameters, producing a compact, specialized policy that improves tool selection accuracy, response quality, and robustness under dynamic query distributions. To address the limitations of fine-tuning proprietary models, we introduce a memory-based middleware that stores query-trajectory-reward tuples and reuses high-reward trajectories at inference time for semantically similar queries, enabling trajectory-level adaptation without modifying underlying model parameters. A learned reuse policy governs whether to replay, adapt, or regenerate trajectories, yielding substantial reductions in inference latency. Overall, the proposed architecture combines GRPO-based policy refinement with learned trajectory reuse, improving accuracy and efficiency while reducing reliance on expensive large-model retraining and preserving deployment flexibility across Agentic AI use cases such as database querying, web search, and multi-tool orchestration. We further validate the framework on a multi-agent information-seeking task involving enterprise email search (Enron corpus) with dynamic redirection to web search via SearXNG, demonstrating generalization beyond incident-management workflows.
Subhadip Mandal, Shalini Mittal, Sukanya Bag et al.· 2026 6th International Confe...· 0 citations
Agentic systems have rapidly advanced in their ability to interact with real-world environments, leverage external tools, and provide services for users. However, unlike natural-world tasks that assume well-defined instructions, human-centered scenarios are characterized by ambiguous requests that lead to large, open-ended solution spaces. Decoding users'personalized preferences is therefore essential for narrowing the candidate solution space. This introduces a new challenge, personalized agentic reasoning, which requires agents to jointly interact with both users and environments to deliver personalized services. In this paper, we present ODYSSE, a Reinforced Fine-Tuning (RFT) framework for personalized agentic reasoning. At its core, ODYSSE proposes Episode-wise GRPO (ESPO), a novel extension of Group Relative Policy Optimization (GRPO) designed to address long action horizons and strong cross-step dependencies in personalized agentic reasoning. Rather than optimizing individual steps independently, ESPO introduces an episode-level reward mechanism together with episodic advantage estimation, enabling upstream evidence to effectively guide downstream personalized decisions and allowing agents to progressively resolve ambiguous user requests across multiple interaction steps. We further propose an episodic batch sampler that groups actions from the same episode into unified training batches, facilitating coherent optimization under ESPO. We evaluate ODYSSE on realistic long-horizon personalized GUI reasoning tasks. Experimental results demonstrate that ODYSSE consistently outperforms both specialist and general-purpose LVLMs, highlighting its effectiveness for personalized agentic reasoning.
Jiaqi Zhang, Tong Chen, Junliang Yu et al.· 0 citations
Group-relative policy optimization has emerged as a key paradigm for training agentic large language models (LLMs) on multi-turn interactive tasks. However, most existing variants fail to distinguish advantages among successful trajectories even when these trajectories differ substantially in their interaction efficiency. For instance, circuitous successes are often assigned the identical outcome reward, causing advantage collapse and severe performance bottlenecks. To this end, we propose Group Planning-aware Policy Optimization (PlanPO), a simple yet effective RL method for learning generalizable planning abilities beyond task-specific high-quality behavior patterns. Specifically, PlanPO introduces coarse-to-fine advantage signals, which capture the relative differences in trajectory-level lengths and turn-level response lengths conditioned on successful trajectories sampled for the same task. Within the group-relative optimization structure, this enables agents to actively learn generalizable and deliberate behaviors spanning interaction planning and textual generation from high-quality rollouts, without degenerating into vanilla length minimization. Experimentally, PlanPO improves over GRPO by 27.2\% on average across the challenging multi-turn benchmarks ALFWorld, WebShop, and SciWorld, outperforming recent powerful baselines while incurring negligible additional training cost.
D. Liang, Liyuan He, Xuan Feng et al.· 0 citations