AUTOMATED CONFIDENCE-ACCURACY GAP PROFILING AND LLM-BASED SAFETY INTERVENTION FOR CONSTRUCTION SITE MANAGERS

Abstract

Construction site managers' safety competency is often undermined by systematic gaps between self-assessed confidence and actual performance. However, conventional safety assessments measure knowledge alone and cannot detect miscalibration. This study presents an automated system that profiles managers' confidence-accuracy gaps and generates training matched to the resulting profile. The system measures both response accuracy and self-reported confidence across three safety domains: regulation comprehension, hazard identification, and risk control. It classifies each manager into one of nine competency profiles defined by accuracy level and calibration status and generates profile-specific training materials using an LLM-based pipeline. In computational validation, the generated material showed robust factual accuracy across two LLM judge models, with accuracy scores of 4.22 from GPT-4.1 and 4.00 from Claude Sonnet 4 on a 5-point scale, while more subjective criteria were evaluator-sensitive. Therefore, profile-specific differentiation was evaluated primarily through an embedding-based analysis, which showed that calibration status drove more content adaptation (mean cosine similarity 0.917) than accuracy level alone (0.952). The principal contribution of this study is a fully automated assessment-to-intervention pipeline that makes profile-specific safety training scalable across organizations with diverse manager populations.