Optimizing Snyk AI Results with Large Language Model (LLM) to Validate False Positive Rate

Abstract

Software vulnerability detection using Static Application Security Testing tools still produces a high number of false positives, which increases the burden of manual verification for developers. A high false positive rate can reduce development efficiency, divert attention from critical vulnerabilities, and indicate a gap between automated detection results and the actual security condition of the code. Addressing this issue is essential to ensure that automated security testing remains a reliable and efficient part of the software development lifecycle. This research proposes the integration of Snyk AI with the Large Language Model GPT-4 as a post-processing validation mechanism based on contextual reasoning. This approach leverages the contextual understanding capability of GPT-4 to re-evaluate flagged vulnerabilities and distinguish genuine security risks from false alarms. The evaluation was conducted on 180 Java source codes from the OWASP Benchmark by comparing the initial detection results with the validation results. Snyk AI produced 53 false positives with an error rate of 29.44%, and after integration with GPT-4, 26 alerts were successfully corrected, resulting in a 49.1% reduction in false positives. These results demonstrate that the integration of GPT-4 effectively improves the quality of software security detection, offering a promising direction for reducing manual verification effort in real-world development environments.