Structurally conditioned counterfactual generation and validation for question answering from scientific text

Abstract

A large language model (LLM)-based pipeline is presented for structured processing of scientific abstracts and for generating paired factual and counterfactual question-answer (QA) instances under document-level intervention constraints. The implemented system extracts entities, events, and relations; induces a lightweight structural causal model (SCM) over scientific variables such as method, data, intervention, outcome, and metric; generates five factual QA pairs per abstract; rewrites them under one selected intervention; and filters inconsistent counterfactuals. The source export contains 100 Scopus abstracts, of which 67 documents with complete pipeline artifacts form the validation cohort. Generation uses gpt-4.1 , while automated validation uses gpt-5-mini under a 15-criterion checklist; three human annotators independently score the same 67 documents. Human agreement is moderate, with a mean pairwise Cohen's kappa of approximately 0.54. Automated validation is high for original QA and relation-level correctness (0.97 and 0.89) but very low for document-level global consistency (0.07), revealing a substantial gap between local correctness and cross-stage coherence. Under the score-based intervention analysis, 25.4% of documents exhibit a validation decrease above the predefined threshold. We refer to this quantity as the intervention degradation rate rather than as bias propagation, because the interventions concern scientific-content variables rather than protected attributes. The study evaluates generation quality, intervention sensitivity, and cross-stage consistency; it does not train a foundation model or claim training-time bias mitigation.