HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software

Abstract

Scientists increasingly want to drive research software by natural-language request, but fluent output helps only if it becomes a correct machine-readable artifact. We introduce HEPToolBench, a benchmark of 28 collider-simulation tasks scored by deterministic, task-specific scorers, plus a three-task structured-debugging extension. We evaluate 42 deployments, from small locally served open-weight models to hosted frontier systems. The central experiment compares direct generation of native HEP-tool syntax with a schema-mediated interface where the model returns a typed representation that deterministic software serializes. Across five matched requests, the mean score rises from 0.418 to 0.902 and task passes from 21/210 to 159/210, with 41 of 42 deployments improving; eleven deployments, seven locally served open-weight models, go from 0/5 passes under native syntax to 5/5 under the structured interface. Values use the corrected v1.2.1 contract, in which the two native scorers that had enforced operational conventions absent from their own prompts are rescored prompt-faithfully cohort-wide. One asymmetry remains: native scorers score the recorded response verbatim and reject Markdown-fenced replies, whereas structured scorers recover JSON from such wrappers; no full-cohort rescoring under a single symmetric extraction rule has been done. An audit of archived responses from 17 locally served deployments shows this does not explain their gains: passes remain six to ten times more frequent under the structured interface when both sides use the same extraction rule. A task pass alone does not guarantee runtime or scientific viability. Within this scope, moving syntax generation into deterministic software can substantially improve reliability for both small local and frontier models. Prompts, scorers, responses, and regeneration scripts are released for independent evaluation and extension.