Conference Open access

KARL: Reinforcement Learning for LLM Agents on Multi-Turn Knowledge-Intensive Agentic Tasks

Xueqiao Sun Xiao Liu Bowen Lv Hanchen Zhang Bohao Jing Zehan Qi Yifan Xu Yuxiao Dong Jie Tang

2026 · Annual Meeting of the Association for Computational Linguistics · pp. 47539-47558 · 0 citations · 31 references

Computer Science

TL;DR

This work introduces KARL (Knowledge-Augmented Reinforcement Learning), a framework that enables LLM agents to dynamically explore structured knowledge sources through multi-turn interactions, and empowers agents to proactively decide when and what knowledge to acquire during task execution.

Abstract

Large Language Models have shown remarkable potential as autonomous agents, but their effectiveness in knowledge-intensive tasks remains limited by passive knowledge utilization. We introduce KARL (Knowledge-Augmented Reinforcement Learning), a framework that enables LLM agents to dynamically explore structured knowledge sources through multi-turn interactions. Unlike existing retrieval-augmented approaches, KARL empowers agents to proactively decide when and what knowledge to acquire during task execution. Our framework incorporates online reinforcement learning with curiosity-driven reward shaping, explicitly in-centivizing knowledge exploration while optimizing tool-use behaviors end-to-end. Extensive evaluation across six structured knowledge benchmarks demonstrates that KARL achieves state-of-the-art performance, with our Qwen2.5-14B-based agent significantly out-performing GPT-4o, Claude-4, and o4-mini on both knowledge graph and database tasks. Source code is available at https://github. com/THUDM/KARL .

DOI Semantic Scholar

Read PDF

Similar papers

VEG: Verbal ϵ -greedy for Semantic Exploration in Multi-Turn RL Agents

This work proposes VEG (verbal ϵ -greedy), a novel framework that leverages external feedback as a dynamic control variable to explicitly balance exploration and exploitation within the semantic space and achieves superior accuracy compared to standard RL baselines.

Yongchang Hao, Jie Hao, Yongsheng Mei et al. · 0 citations

Jul 2026

TAPO: Transition-Aware Policy Optimization for LLM Agents

This work proposes TAPO: Transition-Aware Policy Optimization for LLM Agents, a unified training framework that alternates between policy optimization and transition supervision, and demonstrates that TAPO consistently improves task performance over pure policy optimization baselines.

Cong Li, Pei-Xi Peng, Yisen Zhao et al. · arXiv.org · 1 citation

#artificial intelligence Preprint Aug 2026

When Evidence Shapes Collaboration: Knowledge-Conditioned Topology Generation for Multi-Agent Systems

This work proposes K-GAT (Knowledge-Guided Agent Topology Generator), a neuro-symbolic framework that formulates collaboration topology design as a knowledge-conditioned structure learning problem, integrating external evidence directly into autoregressive graph generation.

Yang-Xiao Jiang, Jia-Run Fan, Ming-Cong Xu et al. · 1 citation

Book Open access Aug 2026

Self-Evolutionary Reinforced Knowledge Distillation for Multi-Modal Tool-Use Agents

Autonomous multi-modal agents are increasingly important in real-world applications due to their ability to reason about complex environments and orchestrate tool use. However, deploying multi-modal large language models (MLLMs) for tool use is often constrained by computational cost and inference latency, creating a p...

Lei Shen, Chengyu Wang, Yuanjie Lyu et al. · Proceedings of the 32nd ACM... · 0 citations

#machine learning Preprint Aug 2026

Learning Generalizable Behaviors for Terminal Agents

River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization is proposed, which achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks.

Yi-Fan Yao, Bo Pang, Xuan-Phi Nguyen et al. · 2 citations

Preprint Aug 2026

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

This work proposes BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively, enabling agents to utilize external skills more effectively.

Tian Pan, Yuan Li, Hong-Da Wang et al. · 1 citation