RAG vs. GraphRAG for Industrial Technical Documentation: A Controlled Evaluation on Laser Manufacturing Manuals

Abstract

Retrieval-Augmented Generation (RAG) enables large language models to answer questions grounded in external documents, yet its effectiveness on specialized industrial documentation remains underexplored. We present, to our knowledge, the first controlled comparison of conventional vector-based RAG and graph-augmented RAG (GraphRAG) on a laser-manufacturing operators manual. Both systems are evaluated on 1,000 queries sharing identical chunking (1,000-character segments, 200-character overlap), embedding (nomic-embed-text), generation (Gemma 3 12B via Ollama), and on-premises hardware (RTX 4090, 64 GB RAM). Retrieval accuracy is low for both architectures: baseline RAG achieves Hit@5 of 32.50% (MRR 0.217) and GraphRAG reaches 34.70% (MRR 0.229). However, GraphRAG improves faithfulness from 71.60% to 86.90% (+15.3 pp), suggesting that graph-augmented retrieval reduces hallucination even when chunk-level retrieval precision improves only marginally. Error analysis identifies four manufacturing-specific failure categories (navigational queries, technical identifiers, command syntax, and figure references) that are absent from general-domain RAG benchmarks. These results indicate that, in this industrial setting, retrieval quality rather than generation capability is the primary performance bottleneck, and point toward hybrid retrieval strategies for safety-critical manufacturing environments.