Analytical performance models --- derivations of throughput or speedup from hardware parameters --- make claims independently verifiable and expose binding constraints, yet rarely accompany architecture papers because building one by hand takes weeks of expert effort. We present Rosetta, a multi-agent LLM pipeline that automatically generates first-principles analytical models from research paper PDFs. Given a paper as sole input, Rosetta produces a mathematical specification, an executable Python model, and a plain-English interpretation --- all autonomously, with zero human intervention. The formalization process itself is the primary value: it surfaces implicit assumptions and identifies missing parameters. Four design decisions address failure modes of na\"ive LLM-based generation: a scientific constitution that prohibits circular reasoning, verify-repair loops with independent critic agents, dual verification separating functional correctness from scientific validity, and a best-of-$N$ ensemble that exploits LLM stochasticity. We evaluate Rosetta across three complementary tracks: expert evaluation of 12 landmark papers (CS1), automated scoring of 97 unfiltered ISCA 2025 and HPCA 2026 papers (CS2), and author self-evaluation by six active research groups (CS3). Across CS1, specification quality scores 4--5/5 on 10 of 12 papers with zero significant hallucinations; across CS2, 56\% of fit-screened papers reach Tier~A insight quality. The strongest finding comes from CS3: Rosetta's output led to revised claims and new experiments in active submissions, and five of six author-evaluators said they would use it again.
Performance modeling is central to hardware design and software optimization, yet constructing these models requires structured reasoning about computation, data reuse, storage, and movement. We introduce PerfReasoning, a benchmark that evaluates LLMs both as direct performance reasoners and as generators of analytical performance-model code. Given workload, architecture, and mapping specifications, models compare mappings and predict off-chip traffic and buffer requirements. The strongest closed-source models exceed 90% on reasoning-based Q&A, and the best open-weight model reaches 82.4%. However, model construction is substantially harder: while GPT-5.6 Sol exceeds 80% pass rate, all other model configurations average below 45% and vary markedly across runs. Task-specific RL raises a 4B model's mapping-reasoning accuracy by 15.7 points, whereas feedback-free multi-round self-revision prompting is not reliably effective. PerfReasoning exposes the gap between plausible architectural reasoning and reliable performance-model construction. We will publicly release the benchmark to support reproducible evaluation and track future progress.
Da Zhao, K. Sankaralingam, Christos Kozyrakis et al.· 0 citations
Gauntlet, an open-source pipeline that analyzes a paper through five independent expert-persona reviewers and an adversarial synthesis stage is studied, and a 98-paper automated ablation shows the gain comes from the multi-agent structure.
N. Aggarwal, A. Dubal, Sreeraj Kannakarankodi et al.· arXiv.org· 0 citations
A general method for adapting among microarchitectural policies as an additional pathway for processor improvement, distinct from structural resizing is suggested.
Yan-Xin Zhang, Ian McDougall, Junnan Li et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
