These results indicate that while CoT-augmented LLMs achieve strong performance on deductive reasoning tasks up to five hops, solver augmentation remains valuable for deeper multihop deduction and for applications requiring robust and verifiable reasoning.
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities in multistep reasoning, yet fundamental questions remain about their ability to perform reliable deductive reasoning. Two contrasting paradigms have emerged: chain-of-thought (CoT) prompting, which treats LLMs as self-contained reasoners, and solver-augmented approaches, which use LLMs as semantic parsers to translate problems into formal logic for symbolic execution. We present a systematic comparison of these approaches under strict single-pass evaluation across three established benchmarks (ProofWriter, PrOntoQA, and LogicalDeduction) and controlled synthetic datasets. Our findings reveal three key insights: (1) on standard benchmarks, solver augmentation yields minimal improvements (
<
1
%
) over zero-shot CoT for state-of-the-art models, suggesting diminishing performance margins between paradigms under current benchmark settings; (2) mid-sized open-source models (14B–32B) match proprietary model performance when solver-augmented, with the primary challenge shifting from generating valid syntax (small models) to accurate semantic parsing (larger models); and (3) on synthetic datasets with controlled difficulty variations, solver-augmented methods demonstrate superior robustness, maintaining 76.7% accuracy retention at extreme reasoning depths (14–17 steps) compared to 62.2% for CoT, and showing greater resilience to distracting facts and rules. These results indicate that while CoT-augmented LLMs achieve strong performance on deductive reasoning tasks up to five hops, solver augmentation remains valuable for deeper multihop deduction and for applications requiring robust and verifiable reasoning.
Reason Popper-ly, a neurosymbolic framework that uses inductive logic programming (ILP) to learn relation composition rules from reasoning traces and deploys them as an online verifier for step-level correction, consistently improves terminal accuracy over standard CoT.
This work presents a theoretical framework that reveals how reasoning steps can amplify error through three failure modes: incorrect sub-task decomposition, incorrect sub-task solving, and incorrect final answer summarization, and introduces structured interventions that adapt CoT generation according to the identified failure types.
Haibo Jin, Peiyan Zhang, Man Luo et al.· Neural Information Processin...· 1 citation
Chain-of-thought (CoT) reasoning improves large language model (LLM) performance while also providing an observable interface to the model's reasoning process. Existing approaches that leverage verbalized CoTs to monitor reasoning correctness, however, largely evaluate the semantic correctness or consistency of individual intermediate steps, rather than how the reasoning process evolves across the trace. As a result, failures distributed across the reasoning trajectory, rather than those localized to a single incorrect step, remain comparatively underexplored. Furthermore, verbalized CoTs need not faithfully reflect the model's internal reasoning, motivating analyses that do not treat individual statements as literal accounts of internal computation. In this work, we therefore ask whether the dynamics of visible CoT can be leveraged to systematically distinguish successful from failed reasoning without assuming such semantic faithfulness. We study a range of LLMs on verifiable Boolean satisfiability tasks with variable complexity, enabling controlled comparisons near each model's capability frontier. Tagging CoT sentences by reasoning function reveals premature verification collapse on SAT problems: incorrect traces enter clause checking earlier, repeat similar operations, and finalize sooner. On UNSAT problems, models presumptuously move towards incorrect SAT conclusions, checking candidate assignments rather than deriving contradictions across constructed cases. Subsequently, a targeted proof-search prompt intervention raises Llama3-70B accuracy from 13.3% to 85%, correcting 84.6% of these errors. These results show that capability failures can manifest as distributed, task-dependent changes in the structure of visible reasoning, and that CoT dynamics agnostic to whether the verbalized trace reflects the model's internal computations can help diagnose and correct failures.
Shashwat Sourav, Aishwarya H. Balwani· 0 citations
SymStep: an LLM makes one atomic claim at a time (DEDUCE: Alice, pet, Cat), then a lightweight constraint propagator checks the claim for consistency with prior accepted deductions, rejects contradictions, and cascades implied facts automatically.
Aida Usmanova, Rui Gao, Dilshod Azizov et al.· 0 citations
It is argued that the next leap in AI4Math systems requires a decisive shift from predefined problem-solvers to research agents that can address frontier mathematical challenges with rigorous formal mathematical reasoning, highlighting core limitations of existing systems in serving as mathematical research agents.
E. Jiang, Xiao Liang, Yikai Zhang et al.· 1 citation
Large Language Models (LLMs) have achieved remarkable success in complex reasoning tasks through Chain-of-Thought (CoT) prompting. However, these models often exhibit"computational overthinking,"generating redundant reasoning steps that increase latency and cost without improving accuracy. Recent studies suggest that CoT trajectories can be significantly pruned, yet existing methods often rely on forcing a static thinking budget, heuristic filtering, sub-optimal early exit via classification, or expensive re-training. In this paper, we introduce OS-Pruner, a lightweight plug-in framework that formulates chain-of-thought pruning as an optimal stopping problem. Given a reasoning prefix, OS-Pruner learns whether further reasoning is worth its token cost by optimizing an explicit utility that trades off final-answer accuracy against generated length. Our novel formulation enables the model to dynamically assess the sufficient point of termination for a reasoning chain. OS-Pruner is designed to be lightweight during both training and inference, and to provide users with fine-grained control over the reasoning-effort vs. accuracy trade-off. On diverse reasoning benchmarks and base models, OS-Pruner achieves 20-60\% reduction in generation length with minimal accuracy sacrifice.
Mohammed Ehab, Aymane El Gadarri, Vivek F. Farias et al.· 0 citations