Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs'ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.
Analysis shows that BRANCH's advantage arises not only from exploring multiple reasoning paths, but also from recovering from truncation: its gains strongly correlate with the baseline rate of empty, budget-exhausted outputs, weakening the hypothesis that different problems require routing among test-time reasoning ope...
Sheng Zhang, Xiao-Min Wu, Xi-Yang Wu et al.· 0 citations
MulTR, a comprehensive benchmark designed for multi-answer temporal reasoning from long unstructured contexts, is introduced, designed to systematically decouple temporal reasoning capabilities from the uncertainty of the number of answers.
Shaojuan Wu· Annual International ACM SIG...· 0 citations
This work proposes a framework that ensembles the reasoning structure, not just the answers, of multiple LLMs by weighted merging of Directed Acyclic Graphs (DAGs) extracted from reasoning chains by weighted merging of Directed Acyclic Graphs (DAGs) extracted from reasoning chains.
Amruta Parulekar, Jinu Lee, Dilek Z. Hakkani-Tür et al.· arXiv.org· 1 citation
This work introduces DocHop, a benchmark for integrated chart--context reasoning in document-style images and constructs DocHop via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, to enable systematic evaluation.
Zhuoran Yu, Le Thien Phuc Nguyen, Jaden Park et al.· 1 citation
A unified taxonomy of 8 categories covering ambiguous and unanswerable questions is addressed, a multi-agent generation pipeline with a two-stage process (NLQ generation followed by SQL grounding) and an explicit Category Conformance validation stage are addressed.
Giovanni Sullutrone, Luca Sala, Sania Aftar et al.· arXiv.org· 0 citations
Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics, rather than tool invocation mechanics.
Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
