It is argued that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.
Abstract
Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants. General purpose benchmarks do not measure whether a model reasons safely and correctly about aviation specific operational knowledge, and the high stakes, regulated nature of the domain makes that gap consequential. We present Pre-Flight, an open source benchmark of 300 multiple choice questions drawn from international standards and airport ground operations material, covering international airport ground operations, ICAO and US FAA regulations, aviation general knowledge and complex operational scenarios. Questions were authored and reviewed by practitioners with experience in air traffic management, ground operations and commercial flying. We evaluate a range of contemporary commercial and open weight models using the Inspect evaluation framework, scoring by accuracy under a standard multiple choice protocol, and we maintain the leaderboard on a rolling basis as new models are released. Against an informal expert reference of around 95%, obtained from a low sample quiz of aviation professionals at a conference, even the strongest model evaluated (released in 2026) reaches 82.7%, having improved only gradually from roughly 75% in early 2025. A substantial and persistent gap below expert level reliability therefore remains. We release the dataset, the evaluation harness and the results, and the benchmark is available within the community evaluations package distributed with inspect_evals. We argue that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.
A safety-gated evaluation framework in which a trajectory succeeds only when all task goals are achieved without violating any hard safety constraint, while safe goal progress and trajectory safety are measured separately is established.
Yuchen Yuan, Zhenghuang Wu, Yuangan Li et al.· 0 citations
This study aims to construct and validate a retrieval-augmented generation (RAG)-driven workflow for automatically generating SJT items and provides preliminary evidence for the feasibility of an automated development pathway for psychological assessment tools based on LLMs and RAG technology.
Yaqian Liu, Qida Hao, Jian Cheng et al.· AHFE International· 0 citations
This work presents an AI-assisted approach that generates candidate hazard scenarios from NASA's Aviation Safety Reporting System (ASRS), and proposes a hybrid variant, conditioning narrative generation on a structured hypothesis produced via evolutionary abduction, improving correctness and reducing variability.
Cristian Mascia, R. Pietrantuono, Daniel Rodríguez et al.· 0 citations
Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management have been semi-automated. In this article, we experimentally evaluate whether large language models (LLMs) can generate operationally realistic ATC transmissions. An experimental general-aviation flight flying over the San Francisco"Bay Tour"route is hand-transcribed and used as ground truth (P0). Through a pilot-in-the-loop process we design five prompt structures (P1-P5) of increasing constraint and embed them in a stateful multi-turn pipeline, where the model plays ATC to a fixed pilot transcript while conditioning on the accumulating dialogue history. Across nine open- and closed-source LLMs we vary the prompt, the presence of a worked transcript from a different experimental flight as an in-context example, and whether the model conditions on its own prior replies or on injected ground-truth history. Turns are scored with lexical, structural, and semantic similarity metrics and by an LLM-as-judge (GPT-5.5) validated against human expert annotation. Supplying a worked example improves similarity, but tightening the prompt does not: the lightest prompts perform best and the most heavily scripted one collapses as its own errors accumulate through the dialogue, which injecting correct history repairs. These results outline a concrete path and its current limits toward LLM-assisted ATC.
Mahyar Ghazanfari, Matthias Casanova, Jordan Kam et al.· 0 citations
A multi-agent, persona-driven prompt generation algorithm generates realistic, multilingual prompts with minimal human review, leaving room for improvement with modern jail breaking techniques in AIR-BENCH Live, a self-evolving successor to AIR-BENCH 2024.
Future exploration EVA operations, especially Mars surface EVAs and some higher-tempo Artemis scenarios, will require greater crew autonomy than the International Space Station paradigm because of communication latency, limited bandwidth, and increased operational complexity. Although large language models (LLMs) and agentic AI systems show promise as onboard decision-support tools, their suitability for safety-critical EVA operations remains unclear due to the lack of domain-specific evaluation frameworks. This paper presents EVA-Bench, a benchmark designed to evaluate foundation model capabilities for exploration EVA (xEVA) support under operationally grounded and safety-relevant conditions. EVA-Bench comprises 651 tasks across six EVA scenario families, three difficulty tiers, and two complementary tracks: Single-Query (SQ) tasks for knowledge retrieval, procedural reasoning, and evidence attribution, and End-to-End (E2E) tasks for multi-step planning, tool use, replanning, and protocol compliance in dynamic mission scenarios. Tasks are grounded in a curated corpus of 84 NASA documents spanning Apollo, ISS, Artemis, EVA standards, and mishap investigations. To jointly assess capability and operational safety, the benchmark integrates a safety-sentinel framework informed by Systems-Theoretic Process Analysis, where critical protocol violations zero the final score regardless of task quality. We evaluate nine models from OpenAI, Google, and Anthropic. Results show that models perform strongly on procedural knowledge retrieval, with top SQ scores above 0.93, but degrade substantially on E2E agentic execution, where multi-step planning and contingency handling remain challenging. Results also show that smaller or mid-tier models can outperform larger models on this domain-specific benchmark, suggesting that targeted training and reasoning design may matter more than model scale alone for safety-critical EVA support. Several models with strong objective performance also trigger safety-critical violations, underscoring that raw capability alone is insufficient for operational deployment. By isolating foundation-model capability from higher-level agentic workflow design, EVA-Bench helps identify which models are most suitable for later integration into EVA decision-support systems.
Kaisheng Li, R. Whittle· 55th International Conferenc...· 0 citations