VisGuard-Ur: A Proof-of-Concept Study on Typographic Jailbreak and Prompt-Injection Attacks Against Urdu-Aware Vision-Language Models

Abstract

Vision-language models (VLMs) are increasingly deployed as document- and image-understanding agents, yet published safety evaluations of these systems are almost exclusively conducted in English plain text. This leaves two attack surfaces largely unexamined for low-resource languages: (1) adversarial instructions embedded as rendered image text rather than typed prompts (“typographic prompt injection”), and (2) the same attack expressed in Urdu, a language spoken by over 230 million people. This paper reports a small, fully reproducible proof-of-concept (VisGuard-Ur) that extends a prior text-only Urdu jailbreak detector (UrduGuard) into the visual modality. We render 30 hand-authored benign and adversarial prompts — in Urdu script, Roman Urdu, and an English control group — into 120 images across four visual variants, build an OCR-plus-classifier defense, and evaluate the full pipeline against a real, locally-run vision-language model (Qwen2-VL-2B-Instruct) rather than a simulated one. Two findings are reported. First, Urdu-script text rendered in Nastaliq — the calligraphic style used in most authentic Urdu print — is substantially harder for both a conventional OCR engine (Tesseract) and the VLM's own text-reading ability than the same text rendered in the straighter Naskh style (OCR character-level similarity 0.40 vs. near-perfect for Latin-script images), identifying the reading stage, not the safety classifier, as the weakest link for this attack surface on Urdu-script inputs specifically. Second, after manually auditing every case the automated judge flagged as a successful attack, we find that image-embedded jailbreak instructions written in plain English produced genuine, explicit policy-violating compliance from the VLM in 4 of 12 cases (33%), while superficially similar Urdu-script and Roman-Urdu attacks mostly produced garbled, non-compliant transcriptions rather than real jailbreaks — the opposite of what the raw automated attack-success-rate number (35%, dominated by Urdu-script false positives) would suggest. Deploying the OCR-plus-classifier detector in front of the VLM reduced the audited system-level attack success rate from 33% to 0% with an 8.3% false-positive rate on benign images. We report this as a small-sample, honestly-scoped proof of concept rather than a benchmark, and detail the dataset size, model, and judge limitations that any follow-up work should address.