Preserving a decade of machine-learning validity across a ground-up refactoring: a five-tier automated validation of RCT-Reviewer, an independent modernization of RobotReviewer
TL;DR
These results establish that reviewers can deploy a maintainable modernization with a clearly bounded performance trade-off, offering a reusable validation framework for clinical NLP modernizations.
Abstract
Evidence-synthesis teams increasingly depend on machine-learning tools to automate risk-of-bias assessment, but these tools frequently rely on unrunnable, deprecated software stacks. Refactoring legacy tools into modern environments is essential for maintenance, yet introduces a critical risk: silently invalidating published performance metrics. Without rigorous validation, systematic reviewers cannot trust that modernized tools retain their predecessors' behavior. We present a five-tier validation framework applied to RCT-Reviewer, an independent ground-up modernization of the widely used RobotReviewer system. To prove preservation, we engineered a compatibility shim to execute the original 2017 code alongside the modernized tool using byte-identical weight files. The framework evaluated predictive validity on a human-labelled benchmark, quantified the cost of dropping obsolete ensemble components, proved bit-exact fidelity, tested parser robustness at scale, and included an external validation arm evaluating agreement on 313 trials against a recent reference standard. A control experiment showing 100% agreement between legacy and modernized implementations on identical text proved that variance versus published values is driven by PDF provenance (open-access versus publisher typeset) rather than codebase refactoring. The modernized tool perfectly reproduced original risk-of-bias judgements in 6,018 comparisons. While the screening classifier experienced a bounded performance reduction (F1 0.969 to 0.925) from removing obsolete components, the model's core mathematical validity is preserved. These results establish that reviewers can deploy a maintainable modernization with a clearly bounded performance trade-off, offering a reusable validation framework for clinical NLP modernizations. Future work will explore retraining obsolete neural network components to close the performance gap.
