Auto-research agents now run machine-learning development unattended for hours, revising data pipelines, models, and validation from their own feedback, yet on most competitions they still finish below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final sub...
The results show that LLM agents, even without executing code, can identify many real-world reproducibility problems from paper-repository pairs, and ReproRepo can serve as a reusable, scalable framework for future evaluations of LLM agents on real-world reproducibility auditing.
Shanda Li, Qiuhong Anna Wei, Jingwu Tang et al.· arXiv.org· 0 citations
This work introduces TraceML, which pairs human and agent work on the same competitions under one version-level schema, and releases the corpus, the schema, the labelers, and the extraction pipeline at https://huggingface.co/datasets/jerryyan/TraceML.
J. Yan, Weiwei Sun, Si-Jie Li et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
