Preprint

BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?

Soumadeep Saha Krish Sharma Akshay Chaturvedi Nicholas Asher

Aug 2026 · 1 citation · 40 references

Computer Science

TL;DR

The nature of test-time exploration in RLVR-trained LLMs is investigated by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence to delineate between entropy arising from stylistic variations and genuine inferential branching.

Abstract

Although reinforcement learning with verifiable rewards (RLVR) has improved the performance of large language models (LLMs) across a variety of reasoning tasks, there is significant debate as to whether RLVR expands the reasoning capability boundary, or just improves sampling efficiency. In this paper, we investigate the nature of test-time exploration in RLVR-trained LLMs by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence. This helps us delineate between entropy arising from stylistic variations and genuine inferential branching. Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy. While RLVR improves adherence to environmental constraints and backtracking capabilities, it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR, albeit at the cost of genuine rollout diversity.

arXiv Semantic Scholar

View source

Similar papers

Jul 2026

Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models

This work presents two converging lines of evidence that linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations.

Antyabha Rahman, Akshaj Gurugubelli, Omar Ankit et al. · arXiv.org · 0 citations

#machine learning Preprint Aug 2026

ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning

A formal analysis showing that joint optimization of the two objectives induces gradient conflict in early training, motivating the sequential design of ERR+, a two-phase RLVR framework grounded in this observation.

Xinle Jiang, Min-Hao Wang, Wen Wu et al. · 0 citations

#machine learning Preprint Sep 2026

Cliff: Learning Process Rewards from the First Mistake

Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout, is proposed and established as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.

Pei-Xuan Han, Runnan Wang, Ketan Ramaneti et al. · 1 citation

#machine learning Jul 2026

Not All LLM Reasoning is Visible in the Chain-of-Thought

This work demonstrates a concrete failure mode where frontier models exhibit invisible reasoning by leveraging semantically irrelevant filler tokens to improve performance on synthetic reasoning tasks and indicates that frontier models already perform consequential computation with no interpretable trace in their outpu...

Vatsal Baherwani, Tom Goldstein, Ashwinee Panda · arXiv.org · 4 citations · ⚡2

#natural language process... Preprint Sep 2026

From Base Rollouts to RL Reasoning: A Budgeted Search Perspective

Results support a qualified internalized-search reading: under the recipe the authors test, much of the measured RL gain corresponds to a change in sampling efficiency toward operating points the base model can already reach under search.

Wen-He Sun, Cun-Xiang Wang, Zi-Jun Yao et al. · 0 citations

#machine learning Preprint Sep 2026

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock ov...

Le-Qi Zheng, Jin-Bo Su, Fang Niu et al. · 2 citations