This study aims to construct and validate a retrieval-augmented generation (RAG)-driven workflow for automatically generating SJT items and provides preliminary evidence for the feasibility of an automated development pathway for psychological assessment tools based on LLMs and RAG technology.
Abstract
In the field of civil aviation, the psychological competency characteristics of cabin crew members are directly related to service quality and flight safety. Although situational judgment tests (SJTs) have proven to be an effective assessment method, their development is costly and time-consuming. The breakthroughs in large language models (LLMs) offer new opportunities for the automated development of assessment tools. Using verbatim transcripts from critical incident interviews with frontline flight attendants as the primary data source, this study aims to construct and validate a retrieval-augmented generation (RAG)-driven workflow for automatically generating SJT items. An expert evaluation approach was employed to assess the quality of items generated by three large models (Model 1: qwen3-14b; Model 2: qwen3-32b; Model 3: deepseek-r1-32b). The results provide preliminary evidence for the feasibility of an automated development pathway for psychological assessment tools based on LLMs and RAG technology, which can significantly improve item development efficiency. However, this study represents an initial exploration, and further research as well as validation through large-scale empirical data are required to optimize and enhance model performance.
It is argued that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.
The correct regulatory interpretation in naval environments is challenging due to the complexity and urgency of decisions based on the International Regulations for Preventing Collisions at Sea (COLREGs). This article presents the development of an intelligent agent named Cognitive Agent for Analysis of Interrelated Problems in Maritime Navigation, hereafter referred to as CAPTAIMN. This agent integrates Large Language Models (LLM) and a Retrieval-Augmented Generation (RAG) architecture to support human decision-making and officer training in safety-critical naval environments. Thus, this work aims to propose a methodology for building an intelligent agent based on LLM and RAG, specifically focused on the assisted and contextualized interpretation of COLREGs. The proposed methodology was evaluated through a quantitative study with 15 maneuvering officers, who assessed 150 responses generated by a local language model using a Likert scale. The results from this phase showed significant approval, with 80% of the responses being rated as 'Agree' or 'Totally Agree' by the officers. These results suggest that the integration of LLM and RAG through CAPTAIMN can provide useful support for both decision-making and tactical training in naval operations.
Gabriel de Sapienza Luna, Arthur Pinheiro de Araújo Costa, Allyson A. da Silva et al.· International Conferences on...· 0 citations
Job Safety Analysis (JSA) and pre-task planning can benefit from prior incident records, yet historical accident data is often stored as unstructured narratives that are difficult to consult at the point of planning. A novel framework centered on large language models (LLMs) for highway construction safety reporting and planning is proposed as a foundation for future agentic applications, prioritizing deterministic, local inferencing. The first aim is to enable classification and quality scoring of incident narratives for existing and future reporting purposes. The second is to evaluate retrieval of relevant historical accidents, related imagery, and trusted industry documents for incorporation into daily safety plans. Neural probes were trained to classify incidents along four multiclass and two binary Occupational Injury and Illness Classification System (OIICS) fields and to derive an overall quality score, evaluated on a test set of over 15,000 narratives and a held-out set of 100 author-labeled records, benchmarked against a majority-vote LLM ensemble. The retrieval of historical accidents, reference imagery, and industry documents was benchmarked across embedding models using standard information retrieval metrics. OIICS classification reached 75% held-out accuracy, though the two binary flags were degenerate. The quality score, while meaningful on one database, was distorted on out-of-distribution fatalities in the held-out dataset. Accident retrieval recovered relevant incidents far above chance, performing best on lexically distinct construction activities. On document question answering, an open-weight decoder embedding model surpassed proprietary models. Overall, this work provides a new framework rooted in local inferencing and text embedding models for future agentic applications, with emphasis on bridging external data to JSA reports.
M. Smetana, Trevor Neece, Lev Khazanovich· 0 citations
This work presents an AI-assisted approach that generates candidate hazard scenarios from NASA's Aviation Safety Reporting System (ASRS), and proposes a hybrid variant, conditioning narrative generation on a structured hypothesis produced via evolutionary abduction, improving correctness and reducing variability.
Cristian Mascia, R. Pietrantuono, Daniel Rodríguez et al.· 0 citations
TableManeuver is proposed, an explainable LLM-based FMR method that reformulates multivariate flight parameter time series as table-understanding inputs and combines recognition accuracy, cross-aircraft robustness, and readable step-by-step reasoning evidence, offering a practical route for applying LLMs to aviation time series analysis.
Liqiang Ren, Haipeng Wang, Xinlong Pan et al.· Entropy· 0 citations
FlightLLM, a prior-guided semantic LLM-based approach for interpretable flight safety analysis that achieves competitive classification performance while generating direct and reasonable explanations for event causes is proposed.