This work study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option.
Abstract
Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on LOGICAL-COMMONSENSEQA and introduce LOGICAL-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR.
Logical reasoning with large language models (LLMs) is a critical capability, as it reflects a system's ability to correctly deduce hypotheses from a given context using faithful deductive processes. However, LLM reasoning has often been shown to be sensitive to small surface-level variations in problem formulation, ra...
Ramya Keerthy Thatikonda, W. Buntine, Ehsan Shareghi· 0 citations
This paper adapts VANESSA, a neuro-symbolic method for chain-of-thought verification, to reasoning-based question answering, and develops a transparent, logic-based approach to answer natural language questions even with phrase variations.
Zacchary Sadeddine, Fabian M. Suchanek· Proceedings of the Thirty-Fi...· 0 citations
A declarative probabilistic framework for pre-training evaluation that makes the semantic structure of model behavior explicit and provides new formal tools for relating evaluation to learning, and links evaluation and learning through a shared semantics.
Kyle Richardson, C. Anderson, Pranav Balakrishnan et al.· 1 citation
MPAR-Bench is introduced, a bilingual English-Chinese benchmark that isolates reasoning breadth through multi-point associative reasoning, and case-level analysis shows that extended reasoning can overturn an initially correct hypothesis.
Si'an Xie, Jiaxu Liu, Biao Yang et al.· 0 citations
Arguments often omit premises or claims, forming enthymemes. We study pairwise logical selection between two candidates for the omitted component. Existing natural language methods can identify or generate candidates but often do not expose how the selected candidate completes the inference, while logic-based approache...
A single encoder trained contrastively to isolate the targeted fact, utilizing synthetic data generated by an LLM and dataset both disjoint from all evaluation settings improves performance on 120 of 126 evaluation settings (95%) spanning 18 model dataset combinations across language and vision-language models.
Joseph Hoche, Quentin Guimard, Gianni Franchi· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
