Skip to content
Open access

Distributional Auditing of Two Large-Language-Model-Augmented Security Pipelines: A Descriptive Enterprise Case Study

2026 · Computers, Materials & Continua · 0 citations · 36 references

Abstract

: Large language models (LLMs) are increasingly integrated into zero-trust security pipelines, but their deployment-specific label distributions remain insufficiently examined. This study develops a distributional auditing approach and applies it to two operational pipelines in a financial-sector enterprise: on-premise Mistral-7B Instruct with 5-bit quantisation and GPT-4o through Azure OpenAI. The pipelines processed disjoint, non-random subsets of Windows Active Directory security events, comprising 22,000 and 146,816 successfully parsed events, respectively. The local pipeline assigned 97.4% of events to Medium, whereas the cloud pipeline assigned 87.97% to High. Cohen’s κ was 0.257 locally and 0.287 in the cloud, with raw agreement of 86.4% and 66.4%, respectively. A seven-component descriptive vector combines output-label prevalence, conditional escalation and buffering indices, and ordinal 1-Wasserstein distance; its two-dimensional projection visualises the observed medium-buffering and high-escalation profiles. These case names describe observed patterns rather than threshold-based classifications. The study contributes an interpretable audit representation and a procedure for inspecting how operational pipelines transform upstream security labels without external ground truth. Important limitations constrain interpretation: only 49% of the pre-processed corpus reached inference, scheduling independence could not be verified, rule-baseline prevalence differed substantially between subsets, and inference configurations were confounded with deployment. Because prompts included rule-layer information, agreement measures represent upstream-to-downstream transition consistency rather than independent classifier agreement. The findings therefore describe the analysed subsets and support neither model-specific causal claims nor generalisation to the enterprise event stream. TrustScore results remain exploratory and non-reproducible. Matched-event replication is required before assessing operational benefit or attributing differences to model identity.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.