AutoJudger : An Adaptive Evaluation Framework for Efficient Benchmarking of MLLMs

Xuanwen Ding Chengjun Pan Zejun Li Ji-Wen Zhang Siyuan Wang Zhongyu Wei Emanuele Tschannen Thomas Bugliarello Unterthiner Daniel Skanda Keysers Fangyu Koppula Adam Liu Alexey Grycner N. Gritsenko Manoj Houlsby Kumar Eli Bingham John Chen M. Jankowiak F. Obermeyer Neeraj Pradhan Theofanis Karaletsos R. Singh Paul A. Szerlip Paul Horsfall Li Cai Kilchan Choi Mark Hansen Lauren Harrell Zhe Chen Weiyun Wang Hao Tian Shenglong Ye Zhangwei Gao Erfei Cui Wenwen Tong Kongzhi Hu Jia-Peng Luo Zheng Ma Jiannan Wu Wenhai Wang Weijie J. Su Guo Chen Sen Xing Muyan Zhong Qinglong Zhang Matt Deitke Christopher H. Clark Sangho Lee Rohun Tripathi Yue Yang Jae Sung Park Mohammadreza Salehi Niklas Muennighoff Kyle Lo Luca Soldaini Jiasen Lu Taira Anderson Erin Bransom Kiana Ehsani Huong Ngo YenSung Chen Ajay Patel Mark Yatskar Mucong Ding Chenghao Deng Jocelyn Choo Zichu Wu Aakriti Agrawal Avi Schwarzschild Tianyi Zhou Tom Goldstein John Langford Anima Anandkumar Khang T. Doan B. Huynh D. T. Hoang T.-H. Pham Quan Nhat H. Pham T. M. Nguyen B. Q. Vo Suong N. Hoang Chaoyou Fu Peixian Chen Yunhang Shen Yulei Qin Mengdan Zhang Xueyi Lin Jinrui Yang Xiawu Zheng S. Galib Shanshan Wang Guan Xu Pascal Pfeiffer Ryan Chesler Mark Landry Sri Satish Wentao Ge Shunian Chen Hardy Chen Nuotian Chen Junying Chen Zhihong Chen Wenya Xie Shuo Yan Shuhao Gu Jialing Zhang Siyuan Zhou Kevin Yu Zhaohu Xing Liangdong Wang Zhou Cao Jintao Jia Zhuoyi Zhang Yixuan Wang Zhe Hu Bo-wen Zhang Jijie Li Dong Liang Yingli Zhao Yulong Ao

· 0 citations · 29 references

TL;DR

This work introduces A 2 -Judger, a novel MLLM-based A gentic instantiation of A uto Judger equipped with semantic-aware retrieval and dynamic memory that significantly improves sample efficiency while maintaining reliable evaluation results.

Semantic Scholar

View source

Similar papers

Book Open access Aug 2026

AutoDavis: Automatic and Dynamic Evaluation Protocol of Large Vision-Language Models on Visual Question-Answering

AutoDavis is introduced, a first-of-its-kind automatic and dynamic evaluation protocol that enables on-demand benchmarking of LVLMs across specific capability dimensions and shows effectiveness and reliability, offering a new paradigm for dynamic benchmarking of multimodal intelligence.

Han Bao, Yue Huang, Yan-Bo Wang et al. · Proceedings of the 32nd ACM... · 0 citations

#artificial intelligence Preprint Aug 2026

MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

This work proposes MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence, which adopts a hierarchical evaluation protocol to measure models' error detection, diagnosis and resolution performance, and applies alignment-aware metrics to assess the coherence between internal reasoning an...

Jin-Zhe Li, Geng-Xu Li, Jinnan Li et al. · 0 citations

#natural language process... Preprint Aug 2026

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

GenRubric is introduced, a self-evolving framework that improves rubric generation from unlabeled queries without requiring additional human annotations during self-evolution, and experiments show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-wri...

Yifan Chen, Hai-Tao Li, Qing-Yao Ai et al. · 3 citations · ⚡1

#machine learning Preprint Sep 2026

Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

This work investigates LLM-based evaluators of natural language generation quality mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and expli...

Himil Vasava, Ming-Zhou Jiang · 0 citations

Preprint Aug 2026

LongChart VQA: A Comprehensive Benchmark for MLLMs with Complex Multi-Chart Reasoning

This work evaluates 10 state-of-the-art MLLMs and examines three factors that influence performance: reasoning patterns, auxiliary tools, and robustness to image perturbations, showing that MLLM accuracy decreases and varies substantially as computational complexity increases.

Ziyan Xiao, Yinghao Zhu, Wen-Ting Zhang et al. · 1 citation

Preprint Aug 2026

Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study

The results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.

Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil et al. · 0 citations