Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writ...
Shan Zhong, Chen-Xuan Xu, Jing-Jie Ning et al.· 0 citations
Scientific names increasingly connect papers from different research topics, leaving readers to recover a paper's identity from context. We study this convergence in 564,861 AI-related arXiv records. Cross-topic name collision grows 4.33-fold from a pooled 2020-2022 reference to 2026, compared with 1.12-fold within top...
This work introduces AgentProofBench, which connects paired tests of a frozen acceptance rule to execution from identical saved openings, and makes reusable verification assessable at both the requirement and policy levels.
Jing-Jie Ning, Xue-Qi Li, Yi Lu et al.· 0 citations
Evaluation budgets in agentic retrieval-augmented generation span questions, search trajectories, and repeated answers. We measure allocation precision, reading efficiency, and cost boundaries using a retrieval-feedback comparison on HotpotQA and MuSiQue. At 34.14--34.39M model tokens, broader question coverage lowers...
The protocol measures predictive credit for research-agent benchmarks and scientific forecasting; natural-explanation credit remained unconfirmed at the tested donor resolutions.
Jing-Jie Ning, Xue-Qi Li, Yi-Bo Kong et al.· 0 citations
Scientific coding agents receive equations, boundary conditions, and output requirements in writing, then must assess the programs they revise. Rules to Tools (R2T) supplies prepared executable checks of public scientific requirements. Matched SciCode repair groups share written checks, starting programs, model, and bu...
Jing-Jie Ning, Guo-Jiang Zhao, Cheng-Xin Xu et al.· 1 citation
Joint reporting of priority responsiveness, presentation-dependent defaults, and task success and cost supports joint reporting of priority responsiveness, presentation-dependent defaults, and task success and cost.
ControlScope compares continuing generated code, editing the next tool call's data arguments, and replacing the unfinished workflow from the same public execution state to evaluate one-time and repeated reviews across filesystem tasks, ALFWorld, and AppWorld.
Jing-Jie Ning, Xue-Qi Li, Yi-Bo Kong et al.· 2 citations· ⚡1
WhatWorkedBench links acquisition, inference, program structure, and delivery through a delivered response surface of configuration scores, which evaluates this experimental understanding through a delivered response surface of configuration scores.
Jing-Jie Ning, Xue-Qi Li, Yi-Bo Kong et al.· arXiv.org· 1 citation
The Discovery Certification Protocol (DCP) turns an outcome claim into an executable audit under a registered model, information boundary, and budget, and a deterministic verifier reproduces these local decisions from frozen records.
Jing-Jie Ning, Shan Zhong, Xiao-Chuan Li et al.· 1 citation
By validating decisions rather than only artifacts, this design turns adaptive search into reusable evidence wherever agents propose executable alternatives against a fixed evaluator.
The sandbox provides a search API that indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN and achieves comparable latency to popular commercial APIs while ensuring stable document rankings across runs.
João Coelho, Jingjie Ning, Jingyuan He et al.· International Conference on...· 3 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.