ISC-Perception: A Hybrid Vision Dataset for Robotic Assembly with Novel Intermeshed Steel Connections

Abstract

Smart and sustainable construction increasingly depends on automation, yet robotic steel assembly still lacks task-specific perception data for bespoke connection systems. The Intermeshed Steel Connection (ISC) is a novel steel connection system that can reduce bolting effort and support faster, more reusable assembly, but dependable perception for ISC-aware robotic assembly remains underdeveloped. No public image corpus exists for ISC components, and collecting real site imagery is constrained by access, safety, privacy, and the limited deployment of ISC in practice. This paper introduces ISC-Perception, a hybrid vision dataset for near-field robotic assembly with novel Intermeshed Steel Connections. The dataset combines photorealistic CAD renders from SolidWorks Visualize, automatically annotated synthetic scenes generated in Unity, and a limited curated set of real ISC and human images. For a normalised 10,000-image Unity-based pipeline example, the proposed pipeline reduces estimated human effort to 30.5 h compared with 166.7 h for manual labelling, while the full training and validation set contains 15,928 images. Detectors trained on the hybrid dataset outperform synthetic-only and photorealistic-only alternatives, achieving mAP@0.50 of 0.756 on the complete test set. A near-size-matched comparison indicates that the improved performance is associated with the hybrid composition rather than dataset size alone under the evaluated training conditions. In a 1200-frame multi-view benchtop robotic assembly experiment, the detector achieves mAP@0.50/mAP@[0.50:0.95] of 0.943/0.823. These results show that ISC-Perception provides a practical route to data generation for emerging construction robotics applications where real imagery is scarce and supports the development of perception modules for robotic steel assembly in smart construction.