Synthesis of Reliable Logic Question Answering Data for Large Language Model with Aid of Operations Research Program
Abstract
Logical reasoning is a highly valued capability in language models. However, high-quality logical reasoning data remain scarce, while manual data creation is costly and time-consuming. We introduce Operations Research Aided Synthesis (ORAS), a solver-verified, two-layer incremental framework for synthesizing logic question-answering data. ORAS leverages operations research techniques to systematically generate high-quality datasets for evaluating large language models (LLMs) and, when used for training, improving their reasoning capabilities. ORAS's pipeline integrates mathematical programming methods to generate question-answer pairs, and then employs a dual-phase approach: the first phase uses optimization algorithms to construct and evaluate the logic systems by solving the problem using a mathematical solver. The second phase translates the logic system to human-readable problems and generates answers according to the information from the mathematical solver. The fine-tuned model achieves a 22%-33% (percentage point) improvement in accuracy on in-sample tests and a 4%-22% (percentage point) improvement on similar out-of-sample tests.
