Domain-Specific Retrieval-Augmented Generation for Metallurgical R&D Knowledge Bases: A Hybrid Graph-Enhanced Approach
TL;DR
This work evaluates a confidence-adaptive graph-enhanced retrieval layer for metallurgical RAG in a controlled synthetic benchmark with explicitly specified generation and evaluation rules and shows how the retrieval rule behaves under controlled conditions.
Abstract
Metallurgical R&D search is difficult for a practical reason: useful evidence is rarely defined by one keyword. A production-support question can depend at the same time on material grade, process route, defect mechanism, property, test method, and numerical conditions. Conventional retrieval-augmented generation (RAG) pipelines largely treat document chunks as independent text and can therefore miss relations that matter for process monitoring, fault diagnosis, and engineering decision support. We evaluate a confidence-adaptive graph-enhanced retrieval layer for metallurgical RAG in a controlled synthetic benchmark with explicitly specified generation and evaluation rules. The benchmark contains 300 generated heterogeneous records derived from a seven-block source distribution and 30 material–process–defect–property archetypes, together with 180 frozen queries: 60 exact, 60 paraphrased, and 60 multi-hop. The main run evaluates robustness to incomplete structured metadata, with 10% missing and 4% erroneous categorical fields. Entity and relation extraction from raw documents is outside the evaluated scope. We compare BM25, TF-IDF, latent semantic analysis, a sparse + dense hybrid, graph-only retrieval, two ablations, and the proposed adaptive hybrid. On the complete query set, the proposed method obtains MRR = 0.992, Precision@5 = 0.980, Recall@10 = 0.859, and nDCG@10 = 0.948. Relative to the sparse + dense hybrid, nDCG@10 increases by 0.186 (24.4%); the paired 95% bootstrap interval is in the range of 0.166–0.206, and the Holm-adjusted Wilcoxon p-value is 2.59 × 10−29. Under severe degradation with 40% missing and 16% erroneous metadata, the adaptive method retains mean nDCG@10 = 0.791, compared with 0.650 for graph-only retrieval and 0.762 for the metadata-independent sparse + dense hybrid. A 5000-run Monte Carlo analysis estimates 6763 chunks and 58.70 MB for indexed vectors plus metadata at a 512-token chunk size and 64-token overlap. The results show how the retrieval rule behaves under controlled conditions; they are not evidence of plant-level effectiveness or of the quality of generated answers. Those questions require external, expert-labeled validation.