Distributional Auditing of Two Large-Language-Model-Augmented Security Pipelines: A Descriptive Enterprise Case Study
Abstract
: Large language models (LLMs) are increasingly integrated into zero-trust security pipelines, but their deployment-specific label distributions remain insufficiently examined. This study develops a distributional auditing approach and applies it to two operational pipelines in a financial-sector enterprise: on-premise Mistral-7B Instruct with 5-bit quantisation and GPT-4o through Azure OpenAI. The pipelines processed disjoint, non-random subsets of Windows Active Directory security events, comprising 22,000 and 146,816 successfully parsed events, respectively. The local pipeline assigned 97.4% of events to Medium, whereas the cloud pipeline assigned 87.97% to High. Cohen’s κ was 0.257 locally and 0.287 in the cloud, with raw agreement of 86.4% and 66.4%, respectively. A seven-component descriptive vector combines output-label prevalence, conditional escalation and buffering indices, and ordinal 1-Wasserstein distance; its two-dimensional projection visualises the observed medium-buffering and high-escalation profiles. These case names describe observed patterns rather than threshold-based classifications. The study contributes an interpretable audit representation and a procedure for inspecting how operational pipelines transform upstream security labels without external ground truth. Important limitations constrain interpretation: only 49% of the pre-processed corpus reached inference, scheduling independence could not be verified, rule-baseline prevalence differed substantially between subsets, and inference configurations were confounded with deployment. Because prompts included rule-layer information, agreement measures represent upstream-to-downstream transition consistency rather than independent classifier agreement. The findings therefore describe the analysed subsets and support neither model-specific causal claims nor generalisation to the enterprise event stream. TrustScore results remain exploratory and non-reproducible. Matched-event replication is required before assessing operational benefit or attributing differences to model identity.