Regime-Conditional Verification (RCV), a lightweight wrapper that adapts an off-the-shelf safety classifier without retraining it, improves adherence to the deployer's policy in every classifier-dataset combination.
Abstract
Safety classifiers deployed with large language models often fail for two reasons: their decisions reflect the policy learned during training rather than the deployer's desired policy, and their performance degrades as deployment traffic evolves. We present Regime-Conditional Verification (RCV), a lightweight wrapper that adapts an off-the-shelf safety classifier without retraining it. RCV estimates, from the classifier's internal representations, the probability that each prediction disagrees with the deployer's policy, and selectively corrects predictions likely to be wrong. The same correctness estimates also provide a label-free signal for detecting distribution shift, enabling a maintenance loop that updates the correctness estimation layer and resorts to classifier fine-tuning only when necessary. Across three off-the-shelf safety classifiers and two benchmark datasets, RCV improves adherence to the deployer's policy in every classifier-dataset combination, catching up to 0.81 of previously missed unsafe content without modifying the underlying classifier. In a deployment study with ten attack campaigns, each a harm category held out of RCV's training, RCV detects every campaign in a dedicated injection panel; in the maintenance census most drift episodes are repaired without updating the classifier, and the fine-tune is reserved for the residual episodes that repair does not restore.
This work proposes the first completely probabilistic architecture-agnostic guardrail to leverage the LLM early output distributional signals for estimating and calibrating the safety probability, thereby enabling early stopping of unsafe ongoing outputs.
Xinzhe Huang, Biwu Yao, Kedong Xiu et al.· 0 citations
Adaptive network intrusion detection systems retrain classifiers after drift alarms, but an alarm detects change; it does not establish that a challenger should replace the deployed incumbent. Promotion is security-relevant because it changes the model responsible for subsequent attack detection, and evaluating it has a methodological problem: promotion conclusions may depend on how the challenger was constructed and on how much evidence supports it. We test that dependence on CICIDS2017, UNSW-NB15 and ToN-IoT with self-contained challenger pipelines, nested candidate-size controls, a common-harness comparison of nine update policies, and a final sensitivity confining every exact feature vector to one evaluation, training or probe role. Incumbent-owned frozen preprocessing amplified apparent promotion harm; with self-contained challenger pipelines the mean full-drift harm did not persist. Raising nominal candidate evidence from 512 to 2,000 samples per class improved promotion under pool-constructed progressive drift by +0.53, +1.67 and +0.38 balanced-accuracy points: positive and statistically resolved in all three benchmarks, but materially benchmark-dependent rather than homogeneous, and driven mainly by fewer false positives. Policy conclusions were partially robust: policy ordering changed with candidate comparability, no policy globally dominated, and earlier compatibility statements for a label-free estimator and a calibrated ensemble narrowed. Validation helped evidence-disadvantaged challengers but added no average benefit at parity. Thirteen replays on real, time-ordered traffic showed no net harm from always deploying. Challenger construction and evidence should be controlled, reported and interpreted explicitly when promotion is evaluated.
Roberto Fernández-Barrios, Iker Pastor-López, A. Pikatza-Huerga et al.· 1 citation
This case shows how evaluation-design validity can be checked structurally before model inference and why base correctness does not determine intervention-response fidelity.
Log Reconstruction and Distance (LoRD), a lightweight post-hoc calibration framework for reliable log anomaly detection, is proposed and demonstrates that LoRD consistently improves confidence reliability and substantially reduces overconfident anomaly-related errors without sacrificing anomaly detection performance.
Cloud policy enforcement in regulated environments produces alert volumes that outpace the human capacity to triage them. Policy-as-Code tools (Open Policy Agent, HashiCorp Sentinel, Kyverno), Cloud Security Posture Management (CSPM) platforms, and Kubernetes admission controllers together generate tens of thousands of policy events per day at enterprise scale, and a substantial fraction are false positives, low-severity findings, or issues already accepted as compensating-control exceptions. The operational consequence is alert fatigue: analysts become desensitised to signals, real high-severity events are missed, and mean time to resolution grows even as the underlying detection stack grows. This paper proposes a risk-prioritisation layer that sits between policy engines and the SecOps triage queue. The layer uses machine learning to (i) suppress or downgrade signals that historical dispositions have shown to be benign, (ii) predict which configurations are likely to drift into non-compliance before they do, and (iii) tune policy thresholds and exception rules from analyst feedback. The paper contributes a three-way task taxonomy for ML-augmented policy enforcement, a four-layer reference architecture that binds signal ingestion, feature engineering, ML scoring, and human-in-the-loop review under an immutable audit trail, a model-selection guide that matches data type (audit log, tabular alert, config drift, multi-signal fusion, feedback stream) to recommended technique, and an evaluation framework that reports both model performance (precision, recall, F1, AUC) and operational outcomes (false-positive reduction, alert volume, mean time to review, missed critical alerts). The paper argues that risk prioritisation is the near-term ML contribution to regulated cloud policy enforcement that clears both the SecOps and the auditor bar, provided the system produces reason codes on every ranking and preserves a compliance-defensible decision-audit store.
Nadeem Siddiqui· International journal of com...· 0 citations
Safe offline reinforcement learning assumes a cost function on every transition. We ask what remains possible when safety can be judged only by comparing short clips and occasionally asking whether an episode exceeded its budget. Certified safety curation answers with a filter-then-clone pipeline: a state-only value trained from segment comparisons scores whole trajectories, Learn-then-Test calibration certifies a selection threshold under a distribution-free $(\alpha, \delta)$ bound on the unsafe fraction of the selection, and behavior cloning follows. We are not aware of prior work certifying the composition of a training set for offline RL or imitation. Oracle controls justify the design: reweighting individual transitions fails even with an exact value, so the value selects whole trajectories. The policies satisfy the cost budget on eleven of fifteen DSRL tasks, one short of cloning the ground-truth safe subset, which needs a label on every trajectory; the uncertified variant reaches twelve. Retrained on the certified selection, the strongest full-label method becomes safe where no setting of its own cost target rescues it. Refusal is predictable: the certificate's probability has a closed form in the purity the pool attains, which the calibration sample estimates and the scorer enters only through.
Adam Haroon, Cody H. Fleming· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
