Three Measurement Hazards in Analyzer-in-the-Loop Repair of Smart Contracts

Abstract

Pipelines that pair a large language model with a static analyzer, feeding findings back as repair instructions, appear throughout recent smart contract repair research. They rest on a rarely examined assumption: that the analyzer output serving as the oracle faithfully records what the analyzer found. I report three ways that assumption fails, identified during a four-contract instrument-validation exercise preceding a planned repair study. First, Mythril v0.24.8 can exit without reaching the analysis phase while returning exit status zero, empty standard error, and a findings array byte-identical to that of a genuinely clean scan; the failure is reported in a sibling JSON field that finding-extraction code has no reason to read. Second, 12 of 23 Slither findings in my validation set fell outside the high, medium, and low impact bands, so an unfiltered count measures a composite whose components may not behave alike under repair. Third, keying finding identity on source location breaks across repair rounds. On the one contract carried through three rounds, location-based keying inflated resolved findings from 7 to 12 and introduced findings from 2 to 7. The underlying instability is established in the warning-tracking literature; my contribution is its consequence for repair metrics, where it biases both transition counts upward and can confound comparison between methods producing differently sized .patches. I separately report an executed exploit showing a specification-level authorization defect that produced no high or medium impact finding. I propose calibration procedures for each hazard and release the harness, contracts, and raw analyzer output at doi:10.5281/zenodo.21586404.