Baselines, Sample Size, and Supervisory Activity in Guarded Reinforcement-Learning Traffic Signal Control: A SUMO Evidence Audit

Abstract

Guarded supervision may leave a learned adviser inactive, obscuring non-inferiority. We audited an eight-version yellowSimulation of Urban MObility (SUMO) program covering three networks and 3480 counterfactual pairs. A contrast model gained 16.38 percentage points of root-mean-square-error skill over zero effect; eight-step history added 0.82 points (95-percent interval −0.58 to 2.26). A 10-percent criterion was met in 10.5 percent of 20-cluster and 24.5 percent of 30-cluster subsamples: finite-pool diagnostics, not historical pass probabilities. A frozen estimator scored 13.99 percent on 1800 fresh pairs after an interim look. In version 3 (V3), 60,079 of 1,123,200 decisions were queried; budget-flagged non-queries confound attribution of gate inactivity. No calibrated V3 lower bound was positive; all 70 executed deviations required an emergency bonus. Under an empirical-Bernstein bound applied post hoc, a 0.10 harm ceiling was arithmetically unreachable below 88 active clusters. In a separate, registered five-network follow-up, a non-learned supervisor over an actuated incumbent reduced stopped time by 8.75 s per scheduled vehicle (12.58 percent; 95-percent interval 6.85 to 10.47 s), but arrivals fell on one network. The follow-up does not validate the audited programyellow. yellowWe recommend reporting baselines, activity denominators, and bound reachability; neither study establishes safety or general traffic benefit.