Prabhjot Singh

Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

This work introduces PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts and proposes the Script Consistency Rate (SCR), which falls as low as 24.8% on this benchmark, as a mandatory metric for script-agnostic evaluation to ensur...