AGORA-Bench: Benchmark for Large Language Model Based Multi-Agent Systems
Abstract
While diverse Large Language Model (LLM) based Multi-agent Systems (MAS) architectures, such as hierarchical pipelines, debate ensembles, and consensus networks claim superior accuracy and robustness over single-model architectures, existing benchmarks conflate these architectural gains with raw model capability. To resolve this evaluation gap, we introduce AGORA-Bench a novel, open-source benchmark explicitly designed to isolate and measure the structural dimensions of LLM multi-agent systems. Grounded in a formal five-dimensional taxonomy, our approach treats architectural properties, communication topology, decision authority, and agent heterogeneity as first-class, tunable variables. We introduce five archetypes, ranging from a centralized baseline to a weighted Byzantine fault-tolerant (BFT) network, and provide an experimental framework for their controlled comparison across tool-augmented, deterministic scenarios. By uncoupling task difficulty from architectural complexity and enforcing strict programmatic grading without LLM-as-a-judge bias, our framework enables rigorous, reproducible ablation studies. Despite its early development state, this benchmark already equips researchers with the empirical tooling needed to conduct initial deterministic comparisons on how, when, and why specific multi-agent coordination protocols succeed.