Beyond Answer Accuracy: Diagnosing Perception Failures in Multimodal Mathematical Reasoning

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance on mathematical reasoning tasks that combine textual and visual information. However, final-answer accuracy offers limited insight into whether errors arise from incorrect perception, modality-dependent information extraction, or downstream reasoning. We introduce a perception-aware diagnostic framework for bimodal mathematical reasoning that represents a model's perceived problem understanding as knowledge graph triples and evaluates them using an Auxiliary Perception Set (APS), a human-verified collection of perception-oriented questions grounded in directly observable input information. We apply the framework to Plane Geometry problems from MathVerse, evaluating three MLLMs across two input representations: text and diagram embedded in a single image, and text and diagram provided separately. We also test controlled textual and visual perturbations to examine robustness under modality-specific noise. Results show that perception accuracy and final-answer accuracy often diverge: models that generate more accurate perception triples do not necessarily achieve higher final-answer accuracy, and the same problem content can produce different behavior depending on input representation. Cross-model triple transfer further shows that downstream performance depends on both the quality of extracted perceptual structure and the model's ability to reason over it. These findings suggest that perception-aware evaluation exposes failure modes hidden by answer-only evaluation and provides a finer-grained diagnostic view of bimodal mathematical reasoning.