Evaluating the Impact of Input Text Properties on LLM-Based Requirements Extraction

Abstract

Large Language Models (LLMs) are increasingly applied to requirements engineering tasks, yet existing benchmarks measure model performance without accounting for how properties of the input text affect extraction outcomes. Among the broad spectrum of requirements engineering activities, this research focuses on requirements extraction, identifying requirement statements in existing documents, a task that can be evaluated against human-annotated ground truth. We argue that measurable text properties can indicate extraction performance. We propose a three-stage research program. Stage 1 defines a metrics toolkit spanning five dimensions (entity coherence, readability, structure, requirements readiness, domain terminology) and validates their discriminative power on a corpus of 376 documents: 19 metrics were used for statistical analysis; all 19 significantly distinguish document types (Welch's ANOVA with Benjamini-Hochberg FDR correction, $p<0.05$), achieving $93.09 \%$ classification accuracy. Stage 2 will assemble a dataset of natural-language documents paired with human-annotated requirement statements, then measure how each document's metric profile correlates with LLM extraction quality. Stage 3 will compare multiple LLMs and prompting strategies to map technique-specific failure boundaries. The envisioned outcome is a decision framework mapping document profiles to expected extraction quality. We report Stage 1 results and the research roadmap.