VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models

Abstract

Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or from both. We introduce VRIQ, a benchmark of 1,391 human-verified IQ-style visual reasoning puzzles spanning five psychometric categories, each instantiated in category-matched abstract and natural-image domains. To study perception and reasoning more directly, we also introduce DiagVRIQ, a 209-item diagnostic subset paired with human-verified, rule-neutral textual descriptions. Across 21 VLMs, the 17 non-frontier models reach a 26.8% unweighted category mean on abstract puzzles, while three 2026 frontier models improve substantially but unevenly, with unweighted category means of 41.3--70.9% on abstract puzzles and at most 78.2% on natural-image puzzles; all remain below human performance. On DiagVRIQ, even when their descriptions are judged correct, the strongest models still answer approximately 9--12% of those items incorrectly. Two interventions, ground-truth description augmentation and tool-augmented inference, improve accuracy, with the largest tool gains on spatially demanding categories, but neither eliminates errors. VRIQ and DiagVRIQ provide a framework for studying visual reasoning through complementary end-to-end, description-based, and intervention-based analyses rather than treating it as a single monolithic capability.