Jun 2026· arXiv.org· Vol abs/2606.26036· 0 citations· 87 references
Computer Science
Abstract
Training-time data poisoning during fine-tuning poses a significant threat to large language models (LLMs) deployed for abstractive text summarization, where small task-specific datasets exert disproportionate influence on model behavior. In this setting, adversaries manipulate fine-tuning data to induce persistent summarization failures, such as biased or harmful summaries, while preserving standard evaluation metrics. We present a unified post-hoc defense framework for detecting and remediating fine-tuning-stage poisoning in summarization models across the machine learning supply chain. Our experiments show that in white-box settings, poisoned document-summary pairs exhibit abnormally high training influence, enabling detection via influence-function analysis with semantic consistency checks. In black-box settings, poisoned models display two to three times greater sensitivity to semantics-preserving perturbations, enabling behavioral auditing without training data access. Beyond existing poisoning formulations, we introduce novel attacks targeting factual distortion and representational bias, showing that poisoning alters summarization behavior without triggering conventional alarms. Across nine architectures and six benchmark datasets under adaptive attacks, our defenses achieve 85-92% detection precision, while gradient-ascent unlearning restores up to 96% of original behavior with minimal utility loss (less than 0.6% ROUGE degradation). These results indicate that fine-tuning-time poisoning leaves persistent structural artifacts, enabling practical detection and post-deployment recovery without full retraining.
D-SCAN (Document-level Signal Collapse Analysis), a lightweight detection framework that monitors attention dynamics to identify attacked generations and proposes D-SCAN, a lightweight detection framework that can detect attacks even when they fail to alter the final answer.
Yingtao Ren, Ziyi Zhao, Yiwei Fu et al.· Annual International ACM SIG...· 0 citations
RAGuard, a layered defense against corpus-poisoning attacks on RAG pipelines, is introduced, showing that keyword-preserving poisons leave lexical retrievers such as BM25 essentially unaffected, an observation that delineates the boundary of the threat model.
This work finds that models cannot reliably reconstruct the original meaning and layer-wise probe classifiers fail to detect the harmful intent of perturbed prompts, and perturbations can occasionally reduce attack success by inducing off-topic or incoherent responses.
This work proposes an explanation-aware safety framework that augments binary harmfulness detection with structured, human-interpretable explanations capturing severity, strategies, trigger spans, ratio-nales, and derived safety factors, and introduces a human–LLM hybrid annotation and canonicaliza-tion pipeline.
Sunghee Dong, Sungwon Yi, Kangmin Bae et al.· 0 citations
This work proposes MemCatalyst, a set of data poisoning tools, aiming to amplify the data auditing performance on VLMs, and forces VLMs to over-learn specific inconsistencies between image features and textual semantics during training, thereby increasing their susceptibility to membership information auditing.
Xukun Luan, Jinyan Liu, Yuhui Gong et al.· 0 citations
Detecting whether a jailbreak attack against a large language model (LLM) has succeeded is a challenging problem in AI security. Existing approaches rely on brittle string matching, expensive LLM-based judges, or task-specific classifiers that often fail to generalize. This paper investigates whether semantic coherence is sufficient to evaluate jailbreak success. In this work, we propose PRISM: Prompt-Response coherence for Identifying jailbreaks via Scoring Method, a lightweight method that models jailbreak detection as a coherence problem between the harmful prompt and the model response. The approach combines multiple semantic similarity signals within an ensemble of classical and simple machine learning models. Training in 3,480 pairs of prompt-responses from AdvBench, HarmBench and Alpaca, PRISM achieves F1 = 0.91 and AUC-ROC = 0.98 in the internal test set. In a held-out validation set of 694 labeled samples never seen during training, it outperforms six baseline attack success estimators, achieving the highest accuracy (0.746) and AUC-ROC (0.840), while being up to $\mathbf{1 6} \times$ faster than LLM-based methods. These results demonstrate that semantic coherence provides a robust and efficient signal for jailbreak detection, significantly reducing the implementation complexity of state-of-the-art approaches.
Costabile Di Gregorio, Lama Sleem, Jérôme François et al.· Annual International Compute...· 0 citations