This work finds that models cannot reliably reconstruct the original meaning and layer-wise probe classifiers fail to detect the harmful intent of perturbed prompts, and perturbations can occasionally reduce attack success by inducing off-topic or incoherent responses.
Large Language Models (LLMs) achieve strong reasoning performance, but their robustness to realistic lexical corruption remains poorly understood. We evaluate four open-weight instruction-tuned models and frontier models across four reasoning benchmarks under keyboard noise, character swaps, and filler insertion. Character-level perturbations substantially degrade accuracy, especially on multi-step reasoning tasks, while filler insertion has little effect. We trace this asymmetry to Attention Diversion: lexical corruption fragments subword tokenization, and the resulting fragments attract disproportionate attention mass, concentrated in middle and final transformer layers. Length-matched controls confirm that fragmentation, not prompt length, drives the loss. A factorial intervention then shows why the damage is hard to undo: fragmentation corrupts token content and attention allocation together, and the two are coupled. Restoring clean attention while the content remains corrupted is actively harmful, restoring content alone is insufficient, and only restoring both recovers a substantial share of the gap. This coupling explains why inference-time strategies, including chain-of-thought prompting, spell-checking, self-repair, and stronger repair models, fail to consistently recover performance: each addresses one channel at a time. Code and data are available at https://github.com/Jiaqian-Janelle/Attention-Diversion
Jiaqi Zhu, Yang Zhang, Junhua Ding et al.· 0 citations
Recent jailbreak attacks demonstrate that large language models (LLMs) can be manipulated to generate harmful outputs through adversarial prompts even after robust alignment. However, prevailing methods typically focus on forcing a desired response at the output layer—a surface-level strategy that is brittle and often fails to bypass the more fundamental safety checks embedded within the model’s internal mechanisms. In contrast, we propose intermediate consistency greedy coordinate gradient (IC-GCG), an approach that extends the GCG framework to explicitly target the model’s internal alignment mechanism. Our method forces a harmful prompt’s hidden representations to mimic those of a semantically related benign counterpart within critical middle layers. By adapting a greedy coordinate optimization strategy to craft an adversarial suffix that minimizes this representational distance, IC-GCG creates “semantic camouflag” that bypasses the model’s internal safety judgments. Experiments on several aligned LLMs indicate that IC-GCG substantially outperforms state-of-the-art baselines, increasing the average attack success rate to 94.3% and proving particularly effective against strongly aligned models such as LLAMA2-7B-CHAT, where it boosts success by over 56 percentage points. Furthermore, the resulting suffixes display superior transferability to other models. Ablation studies confirm that this high efficacy critically depends on a strong semantic link between the harmful and benign prompt pair, empirically validating that the integrity of intermediate representations is a crucial vulnerability and highlighting the need for a new class of defenses that operate directly on the model’s internal states.
Zichu Ren, Donghai Zhu, Haibo Hong et al.· IEEE Internet of Things Jour...· 0 citations
The rapid advancement of Large Language Models (LLMs) makes it increasingly difficult to distinguish human writing from machine-generated text. Training-free detection offers a scalable solution, yet common confidence-based metrics mainly measure average token probabilities and often miss the signal fluctuations that characterize human writing, which we call"generative vitality". Spectral analysis offers a way to capture this vitality, but its mechanism and practical boundaries remain underexplored. In this paper, we analyze spectral detection from both theoretical and empirical perspectives. We connect spectral energy to variance in proxy log-probability trajectories and explain how broader human token choices create the fluctuations used by frequency-domain indicators. We further show that the strength of this signal depends on text length and sampling range: spectral evidence is clearest for long, continuous, constrained generation, while short, fragmented, mixed, and edited settings require complementary confidence and fluctuation views. These findings clarify when frequency-domain detection works and provide guidance for future multi-dimensional detector design.
Haitong Luo, Xuying Meng, Weiyao Zhang et al.· 1 citation
This work proposes TinyAttack, a novel adversarial attack framework designed to exploit vulnerabilities in LLMs through Unicode-based stylistic transformations, which utilises five Unicode variants to modify the visual rendering of text without altering its underlying semantic or syntactic structure.
M. Mamta, B. Grecu, O. Cocarascu· Annual Meeting of the Associ...· 0 citations
Large language models (LLMs) are increasingly released as open-weight models with safeguards against harmful requests. Nevertheless, sentence completion remains vulnerable to incomplete harmful prompts. In this work, we formalize this phenomenon as incomplete prompt jailbreaks (IPJ) and provide a systematic empirical characterization of when and how incomplete prompts elicit harmful continuations. We analyze diverse attractor types associated with incomplete sentence continuation and show that LLMs systematically delay refusal until sentence termination. We further demonstrate that training models to refuse incomplete harmful prompts via parameter tuning is insufficient, failing to generalize across both content domains and attractor types. To enable fine-grained control, we identify two functional neurons: termination and continuation neurons. By clarifying their roles in sentence completion, we highlight the potential of neuron-level interventions for more precise and robust IPJ defenses.
Y. Kim, Bumjin Park, Jaesik Choi· Annual Meeting of the Associ...· 0 citations
Large language models (LLMs) are increasingly being used as automated judges for relevance evaluation in information retrieval, yet their robustness to adversarial manipulation remains insufficiently understood, particularly in multilingual settings. In this work, we investigate the impact of cross-lingual prompt injection attacks on LLM-based relevance judgments using TREC Deep Learning collections and two open-weight models under established prompting frameworks. We examine both instruction-based and content-based injection strategies in 8 languages spanning different resource levels. Our results demonstrate that multilingual query-based injections are highly effective in inflating relevance scores while simultaneously evading existing prompt-injection defenses. We further found that, although existing defense mechanisms can be modified to mitigate such attacks, these injections can be easily adapted to bypass them. These findings highlight a critical gap in current defense approaches and demonstrate that language generalization can act as an attack vector, underscoring the need for more robust and proactive evaluation frameworks for LLM-as-a-judge systems.