As Large Language Models (LLMs) become the mainstay for information retrieval and summarization tasks, ensuring that they are always non-partisan and invulnerable to political bias is a critical step towards safer and more trustworthy Artificial Intelligence (AI). Current model alignment paradigms, such as reinforcement learning from human feedback (RLHF), make LLMs follow overarching safety instructions. However, this instruction tuning can be exploited via adversarial prompt injection and be used to generate unsafe content. In particular, political bias has not been specifically targeted by modern alignment techniques as harmful and biased content. To address this vulnerability of LLMs, we propose mitigation strategies using Chain of Thought (CoT) prompting and Direct Preference Optimization (DPO). Using a public dataset of legislative videos, we generate summaries using LLMs, inject bias via adversarial prompting and evaluate their performance on a four axis scale designed for political summarization. In this paper, we present different methods to shield LLMs against the injection of political bias. Our results demonstrate that the proposed Recursive Self-Correction approach raises model performance from a Political Neutrality Likert scale baseline of 2.14 to 4.56, averaged across all models, demonstrating effective inference-time mitigation of political bias in LLM-generated summaries.
Large language models (LLMs) are increasingly being used as automated judges for relevance evaluation in information retrieval, yet their robustness to adversarial manipulation remains insufficiently understood, particularly in multilingual settings. In this work, we investigate the impact of cross-lingual prompt injection attacks on LLM-based relevance judgments using TREC Deep Learning collections and two open-weight models under established prompting frameworks. We examine both instruction-based and content-based injection strategies in 8 languages spanning different resource levels. Our results demonstrate that multilingual query-based injections are highly effective in inflating relevance scores while simultaneously evading existing prompt-injection defenses. We further found that, although existing defense mechanisms can be modified to mitigate such attacks, these injections can be easily adapted to bypass them. These findings highlight a critical gap in current defense approaches and demonstrate that language generalization can act as an attack vector, underscoring the need for more robust and proactive evaluation frameworks for LLM-as-a-judge systems.
Multimodal Large Language Models (MLLMs) often suffer from modality bias, where the model disproportionately relies on one modality while neglecting critical information from others. Existing debiasing methods via modality masking create biased responses by completely removing an entire modality, forming an extreme and static training environment. However, real-world multimodal bias often emerges under subtle perturbations (e.g., mild occlusion, noisy instructions), where both modalities are present but the model is tempted to rely on spurious shortcuts. We propose B-APO (Bias-Targeted Adversarial Preference Optimization), which casts debiasing as a bias-targeted min-max game: we generate hard negatives by applying small adversarial perturbations in the latent space to maximally induce language-vision-prior reliance, and then perform preference alignment to enlarge the margin between clean and adversarial responses. This encourages the model to anchor on true cross-modal evidence even under the most adversarial conditions. Extensive experiments on bias and hallu-cination benchmarks demonstrate that B-APO achieves superior debiasing performance while maintaining general capabilities.
Pinlong Zhao, Zike Ding, Zengshu Ye et al.· Annual Meeting of the Associ...· 0 citations
Membership inference attack (MIA) has emerged as a promising tool for auditing the training data of LLMs, supporting data privacy and copyright protection. Most existing MIA methods rely on the assumption that LLMs assign higher confidence scores to training samples than to non-training ones. However, since LLMs generate text by sampling high-confidence tokens, they naturally produce AI-generated texts (AIGTs) that also satisfy this assumption. In this work, we empirically confirm that such AIGTs, regardless of whether they are generated by the target LLM, can lead existing MIAs to assign even higher membership likelihoods than those of true training samples, thereby significantly un-dermining their reliability. To address this challenge, we propose a robust membership inference framework for reliably identifying training data. Our method adopts a mixture-of-experts formulation to jointly model interactions across complementary features derived from multiple MIA methods and AIGT detectors, which can remain robust against ad-versarially generated samples. Furthermore, by leveraging expert components, our method provides explainable insights into the characteristics of member data. Experiments on various datasets and LLMs show that adversarial samples substantially degrade the performance of baselines, whereas our method preserves performance close to that of the unattacked setting. Codes and datasets are released at https://github.com/kong-hyh/MoMIA .
Yuanhong Huang, Huili Wang, Xueying Bai et al.· Annual Meeting of the Associ...· 0 citations
Large Language Models (LLMs) are increasingly used in settings where the original user prompt may be unavailable, redacted, or untrusted. This makes response-only adversarial detection an important but challenging problem, because the detector observes only the generated response while the label is determined by the originating prompt. In this study, we investigate response-only adversarial classification of LLM outputs through a comparative evaluation of lexical, contrastive, and contextual representations. Specifically, we compare TF-IDF features, triplet-loss-based metric embeddings, TF-IDF+triplet concatenation, and a Sentence-BERT contextual embedding baseline across three tasks: Prompt Type, Adversarial Type, and Adversarial Class. Experiments on 158 592 response records show that response-only detection is feasible but substantially more difficult than prompt-aware detection. Under stratified group 10-fold evaluation, the strongest TF-IDF+XGBoost baseline achieves 72.37% accuracy for Prompt Type, 82.06% for Adversarial Type, and 56.40% for Adversarial Class. Leave-one-LLM-out evaluation further shows that cross-model generalization remains challenging. Triplet-based metric learning provides model-dependent benefits, particularly for some Adversarial Type configurations, but it does not uniformly outperform strong lexical baselines. Low-dimensional embedding analysis shows that Insertion samples are relatively separable, whereas Prefix and Suffix samples substantially overlap. Overall, the findings provide a realistic benchmark for response-only adversarial detection and clarify the limits of contrastive representations when prompt information is absent.
A. Nasution, Ahmet Emre Ergün, Aytu˘g Onan et al.· IEEE Access· 0 citations
Large Language Models (LLMs) often produce outputs that reflect social biases, toxicity, or unfair treatment of demographic groups, undermining trust and fairness. While prior mitigation strategies frequently rely on complex architectures, access to model internals, or costly fine-tuning, we argue that simplicity can be a strength. We introduce StarDTox, a lightweight, critique-and-revise multi-agent framework that leverages the LLM's own internal knowledge, via a small number of coordinated prompts, to self-correct harmful outputs. Dedicated agents independently assess bias and overall output quality, and their feedback is integrated to guide prompt-based revision. Without modifying model weights or requiring any extra finetuning, StarDTox offers strong bias mitigation and high-quality outputs across both open-ended text generation and structured tasks, outperforming other baselines. For the text generation task, on the RealToxicityPrompt dataset, it reduces toxicity by over 50% compared to other baselines, while maintaining over 90% fluency. In addition, in structured tasks, on the BBQ benchmark, it achieves the lowest bias scores across both ambiguous and disambiguated examples, without sacrificing accuracy.
Shirin Tahmasebi, Narjes Nikzad, A. H. Payberah et al.· Annual International Compute...· 0 citations
With the rapid deployment of neural machine translation (NMT) and large language models (LLMs), AI-assisted translation has become a cornerstone of multilingual communication. Despite achieving impressive fluency, these systems often perpetuate subtle yet systematic cultural biases embedded in training corpora, model architectures, and inference pipelines. This paper presents a systematic review of cultural bias in AI translation, organized around three research questions: (1) how cultural bias manifests, (2) how it can be identified, and (3) how it can be mitigated. Drawing on recent advances in machine translation, multilingual NLP, and AI fairness, this study analyzes manifestations across gendered stereotyping, religious oversimplification, regional framing, and cultural normalization; and then synthesizes detection methods, including benchmark-based evaluation, contrastive probing, embedding association tests, and human-in-the-loop assessment. For mitigation, this paper proposes a five-layer framework spanning data auditing, model adaptation, inference-time intervention, post-editing, and governance. To validate the framework, we conduct five proof-of-concept experiments: cross-lingual gender bias detection with statistical testing, systematic cultural fidelity evaluation under prompt engineering, contrastive sentiment analysis under high-/low-risk contexts, word embedding association tests (WEAT) with permutation-based significance, and an integrated audit pipeline with automated mitigation. Results demonstrate significant gender bias (χ²=29.99, p<0.001), a pervasive "male-as-default" phenomenon, significant gains from culture-aware prompting (p=0.03), and robust embedding-space bias (permutation test p=0.0001). The audit pipeline successfully integrates detection and mitigation into an actionable workflow. We conclude by outlining future directions for low-resource languages, intersectional bias, and production-level deployment.
Dongyao Zhang· Applied and Computational En...· 0 citations