AgentCity: An AI-Maintained Continuous Benchmark for Traffic Prediction

Abstract

Traffic prediction is a fundamental component of intelligent transportation systems, and recent research has explored a wide range of prediction tasks and modeling approaches. While several benchmarking frameworks have been proposed to support fair and reproducible evaluation, most existing benchmarks rely on manual maintenance, making it difficult to continuously integrate new models and evaluate them under a unified set of data processing and evaluation protocols as the literature grows. In this work, we present AgentCity, an AI-maintained framework for the continuous construction and evaluation of traffic prediction benchmarks. AgentCity formulates benchmark maintenance as an automated, agent-driven workflow that supports literature retrieval, model and data integration, and standardized evaluation under unified protocols. Built on top of AgentCity, we release a publicly accessible traffic prediction benchmark1 covering four representative tasks, including traffic state prediction, trajectory location prediction, estimated time of arrival prediction, and map matching, and aggregate 74 representative models evaluated across multiple datasets under consistent evaluation settings, together with task-wise leaderboards and detailed evaluation records. We further validate the reliability of AgentCity through benchmark validation studies on reproduction fidelity and consistency across different code-oriented agents. By automating key stages of benchmark maintenance, AgentCity supports the continuous integration and evaluation of traffic prediction models under unified protocols. 1Project website: https://www.agentcity.city/ Source code: https://github.com/Beihang-BIGSCity/AgentCity.