Assessing Static Analysis Tools for Security Vulnerability Detection: An Empirical Study

Abstract

Static analysis tools are widely adopted to support security vulnerability detection in modern software development, particularly when integrated into continuous integration (CI) pipelines. However, the increasing number of available tools and the operational constraints imposed by CI environments complicate tool selection, while empirical evidence supporting informed adoption decisions in realistic CI settings remains limited. This study addresses this gap by empirically assessing static analysis tools for security vulnerability detection in Java projects executed within GitHub Actions pipelines. The evaluation considers a selected set of representative open-source tools and analyzes their behavior across repositories containing documented, real-world vulnerabilities. The assessment focuses on detection effectiveness, coverage of vulnerability categories, consistency and complementarity across tools, and operational behavior in CI, including execution overhead and stability. The results reveal substantial variability among the evaluated tools, demonstrating that no single solution simultaneously optimizes detection capability, execution efficiency, and CI robustness. These findings expose explicit trade-offs that directly affect the practical adoption of static analysis in CI-driven development workflows. By providing CI-grounded empirical evidence, this study supports more informed tool selection and configuration decisions, benefiting researchers, practitioners, and organizations seeking to strengthen security-oriented practices in continuous integration environments.