From Jargon to Action: Evaluating Open-Weight LLMs for Plain Language Summaries of Pentest Reports

Abstract

Penetration testing reports are a critical artifact in the cybersecurity workflow, yet their technical complexity frequently limits their utility for non-specialist stakeholders involved in risk remediation decisions. This paper investigates the feasibility of using four open-weight large language models, DeepSeek-r1:32b, Qwen3.5:35b, Gemma4:31b, and GLM-4.7-flash:32b, to generate plain-language summaries of penetration testing reports. A corpus of 65 publicly available pentest reports was used for evaluation. Model outputs were assessed across four dimensions: readability, technical term density, semantic similarity to the source document, and factual correctness using LLM-as-a-judge evaluation. Two classical extractive methods, LSA and TextRank, were included as baselines. Readability analysis using seven established metrics showed that Qwen3.5 and Gemma4 produced the most accessible summaries, reducing mean Flesch Reading Ease scores from 25.2 in the originals to 49.2 and 52.1 respectively, and lowering grade-level scores from post-graduate to high-school equivalents. Results across the remaining evaluation dimensions further indicate that appropriately selected open-weight LLMs can produce accessible and factually grounded summaries of technical security documents, offering a practical alternative to proprietary solutions in privacy-sensitive deployment contexts.