A novel pipeline and benchmark for automated technical datasheets processing
TL;DR
The findings quantify the trade-offs between human effort, automation, and large language models performance, providing a data-driven analysis of modern models’ capabilities and limitations in handling the unique complexities of industrial documentation and offering a methodology for developing more scalable and reliable solutions for semantic extraction from industrial documentation.
Abstract
Technical datasheets are fundamental for manufacturing tasks like design, engineering, procurement, and maintenance, but their varied, unstructured PDF formats make them time-consuming to compare and their data remains largely unexploitable by automated systems. This lack of data accessibility necessitates manual processing, which is both time-consuming and error-prone, ultimately reducing the overall usability of the information. Although recent research has addressed knowledge extraction from manufacturing process specifications and semi-structured web sources, no methods effectively overcome the limitation of extracting data from highly heterogeneous, unstructured PDF technical datasheets. Automating this process involves challenges that go well beyond conventional text extraction, requiring advanced semantic modeling to accurately identify, classify, and interrelate product attributes within structured and hierarchical formats (e.g., JSON-based schemas). The difficulty is further amplified in multi-product catalogues, where contextual ambiguity and interdependencies increase significantly. Current benchmarks do not adequately address these challenges, as their limited metrics fail to reflect the depth of semantic understanding necessary for industrial applicability. This paper seeks to address this gap by introducing a highly adaptive automated extraction pipeline and systematically comparing its performance with two alternative extraction strategies: a human-assisted annotation workflow and a zero-shot approach: Using three representative datasets, covering heterogeneous single products, multi-product catalogues, and a homogeneous single-company archive, we systematically benchmark each method’s accuracy, efficiency, and robustness. Our findings quantify the trade-offs between human effort, automation, and large language models performance, providing a data-driven analysis of modern models’ capabilities and limitations in handling the unique complexities of industrial documentation and offering a methodology for developing more scalable and reliable solutions for semantic extraction from industrial documentation.