Artifacts to Actors: Assessing the Maturity of Software Supply Chain Security Measurement Practices

Abstract

Software supply chain attacks are an increasing threat. Industry reports from vendors, and open-source foundations play an important role in shaping how practitioners understand these risks and allocate their security testing budget. However, it remains unclear which aspects of supply chain security these reports measure and which they overlook. To address this gap, we conduct a systematic gray literature review of 37 reports published between 2021 and 2025, identified through Google searches in two countries and LLM-assisted discovery. Two reviewers independently screened the reports and extracted their metrics, classifying each by supply chain concept and measurement focus: Technical Artifact (TA), People & Practices (PP), or Practitioner Sentiment (PS). At the report level, 59.5% of reports contain a majority of TA metrics. Security vendors account for 46.9% of publishers and devote 66.2% of their metrics to TA, a pattern we describe as the “vendor lens effect.” We also find that several areas central to security testing practice, including security requirements specification and threat modeling, are absent or underrepresented. These findings indicate that industry measurement remains largely tool-driven, emphasizing artifact properties that can be readily automated. More holistic assessments will require practitioners and researchers to develop and validate complementary measures of people and practices.