Same Words, Different Procedure: Counterfactual Transfer Tests for Reusable LLM-Agent Skills (Software and Reproducibility Artifacts)

Abstract

This versioned archive contains the implementation, frozen protocols, native interaction records, tests, audits, analyses, and manuscript sources for the Counterfactual Skill Transfer Test (CSTT). CSTT is a state-matched six-condition protocol that crosses source procedural compatibility with lexical similarity and includes no-skill and binding-only controls. The frozen confirmation contains 60 zero-overlap ALFWorld targets, three open local model families, 180 paired target-model blocks, and 1,080 native episodes. Matched-far skills achieved 97.8% native success versus 10.0% for lexically near procedural lures, an 87.8-point paired gain (target-stratified 95% CI 82.8-93.3; task-clustered randomization p = 1.00e-6). Relative to binding-only, the net gain was 83.3 points with zero observed harmful discordances. The archive includes integrity manifests and CPU-only verification scripts. Third-party ALFWorld data and model weights are not redistributed; pinned identifiers, revisions, paths, licenses, and integrity checks are provided instead.