Benchmarking Prompt Injection Attacks on LLMs: Turkish Vulnerability Assessment and English Comparative Analysis

Abstract

Large language models (LLMs) are increasingly deployed in multilingual settings, yet their safety behavior under Turkish harmful prompts and prompt injection attempts remains insufficiently characterized. This study evaluates the adversarial robustness of 55 open- and closed-source LLMs under paired Turkish and English harmful prompt conditions. We constructed a benchmark of 790 Turkish adversarial prompts, translated the prompts into English for cross-lingual comparison, and applied both prompt sets to the model pool. Model responses were labeled as harmful, harmless, or hallucinatory, and safety was analyzed using safety scores, Turkish–English ranking differences, and inter-rater reliability based on Fleiss’ kappa. The results reveal substantial variation across models. Closed-source systems generally achieved higher safety scores and stronger filtering behavior, whereas open-source and Turkish-oriented models showed a wider performance distribution. GPT-5.4 ranked first in the Turkish tests with a 99.37% safety score but decreased to 96.71% in the English tests, while Qwen3.5:27B ranked first in English with 97.47%. These differences suggest that safety mechanisms are not fully language-invariant. Hallucination also emerged as a distinct safety risk, particularly in Turkish evaluations. The findings indicate that Turkish LLM safety cannot be inferred from general model capability alone and should be assessed through language-specific, culturally aware, and continuously updated adversarial benchmarks.