An LLM-based CI/CD analysis pipeline that combines repository enrichment, anti-pattern detection, stage mining, and recommendation generation over a large GitHub corpus is presented, arguing for CI/CD observability that combines diagnosis, context, and human review.
Abstract
CI/CD workflows have become executable operational policy: they decide what gets built, tested, released, and deployed, and they mediate how maintainers interact with delivery infrastructure. That makes them an important measurement point for cyber-systems engineering. Recent large language model (LLM) work shows that workflow stages can be recognized directly from configuration files, but stage labels alone do not tell us whether a workflow is brittle, unusual for its ecosystem, or worth revising first. We present an LLM-based CI/CD analysis pipeline that combines repository enrichment, anti-pattern detection, stage mining, and recommendation generation over a large GitHub corpus. Starting from 59,550 repositories with at least 1,000 stars, we identify 34,225 projects with CI/CD and collect 127,559 configuration files. Across 75,201 analyzed workflows, the anti-pattern detector reports 434,769 findings, dominated by reliability and maintainability issues. Across 59,906 configurations, stage usage differs significantly by language ($\chi^2 = 4168.88$, $p<0.001$, Cramer's $V = 0.063$), and domain analysis shows distinct operational profiles, including higher release and cache usage in mobile projects. For repository-level recommendation generation, few-shot prompting performs best overall, averaging 8.25 recommendations per repository with 96.1% YAML-valid snippets. Taken together, the results argue for CI/CD observability that combines diagnosis, context, and human review rather than treating workflow mining as a stage-classification problem alone.
This study investigates whether a frontier LLM can generate Dockerfiles and Docker Compose configurations for multi-service applications using repository contents without access to developer-authored deployment artifacts and analytically derives a minimal explicit deployment specification for information that cannot be reliably inferred from repository artifacts.
Oleg Grynets, Kyrylo Fursov, V. Lyashkevych et al.· arXiv.org· 1 citation
Continuous Integration and Continuous Delivery (CI/CD) pipelines accelerate software evolution but complicate the timely assessment of structural degradation. While code smells effectively indicate maintainability risks, their detection remains largely confined to isolated static analysis rather than being actively integrated into continuous workflows. To address this gap, we propose an explainable AI-driven approach for code smell classification designed specifically for software quality risk analysis in CI/CD environments. Using structural metrics related to complexity, coupling, and cohesion, we train a Random Forest model to classify four prevalent smells: Blob, Long Method, Feature Envy, and Data Class. Evaluated on labeled Java systems, our approach successfully captures the structural patterns indicative of software decay, achieving 82% accuracy and a 0.86 macro F1-score. Beyond raw performance, we operationalize these classifications as interpretable risk indicators within a prototype CI/CD pipeline, prioritizing human-centered evaluation. Ultimately, this work advances the AI for Software Engineering (AI4SE) domain by demonstrating how explainable machine learning can practically augment continuous quality assurance in agile development.
J. Domingues, D. Duarte, António Sousa et al.· International Conference on...· 0 citations
Deploying large language models for AI-driven workflow generation in a production enterprise platform is benchmarked across 29 real-world IT automation scenarios, two generation pipeline architectures, and eight independent runs per prompt-model-pipeline configuration.
Xavier Wrenn, Radoslav Raykov, Aleksandar Angelov et al.· 0 citations
The rapid evolution of large language models (LLMs) has introduced new challenges in model development, deployment, monitoring, and governance. Traditional software-focused Continuous Integration (CI) pipelines are insufficient for managing the iterative and data-intensive lifecycle of LLMs, which require continuous data validation, model retraining, bias and safety auditing, reproducibility checks, and scalable deployment. This paper proposes a comprehensive CI pipeline architecture tailored to the unique requirements of LLM lifecycle management. The framework integrates automated data quality assessment, modular training workflows, version-controlled model artifacts, continuous evaluation against multi-dimensional metrics, and responsible AI checks including fairness, robustness, and alignment. We discuss implementation patterns using modern MLOps tooling, highlight operational challenges, and present best practices for ensuring reliability, traceability, and ethical compliance in LLM-centric systems. The proposed approach facilitates faster iteration cycles, safer model updates, and more efficient long-term governance of LLM deployments.
H. Mohamed· International Journal of Art...· 0 citations
We study whether LLMs can produce executable workflow artifacts that remain consistent across graph structure, tool implementation, schema bindings, and runtime wiring. In this setting, correctness depends on cross-layer consistency: a workflow may be structurally plausible, yet still fail because tool implementations, schema bindings, or runtime execution do not align. Existing benchmarks largely evaluate these capabilities in isolation or rely on trajectory-level proxies, leaving open whether generated workflow artifacts execute end-to-end. We introduce \textbf{MCPGen}, an executable benchmark for Model Context Protocol (MCP) workflow development. MCPGen contains 100 self-contained MCP projects across 16 application domains and evaluates three diagnostic tasks: workflow reconstruction, tool creation, and backward-compatible workflow extension. We evaluate 11 representative LLMs in a single-turn foundation-model setting, assessing generated artifacts through static analysis, unit and integration tests, and process-isolated end-to-end execution. Models reach 88.5\% on workflow reconstruction, but no model exceeds 57\% end-to-end execution success. Per-tool unit-test pass rates reach 63.8\%, while project-level integration success does not exceed 45\%, suggesting that integration remains a major bottleneck even when isolated tool tests pass.
Yingxuan Yang, Jia-Qi Liu, Li-Rui Guan et al.· 0 citations
With more than 20,000 Common Vulnerabilities and Exposures (CVEs) reported annually, software vulnerabilities represent a critical cybersecurity challenge. This volume has intensified the demand for automated detection and analysis, motivating the integration of large language models (LLMs) for such tasks. However, existing vulnerability benchmarks are not suitable for evaluating LLMs' capabilities in vulnerability assessment, as most of them 1) rely on narrow data sources, 2) lack deep context, and 3) focus on single-turn Q&A rather than realistic, multi-stage analyst workflows. To address this gap, we introduce OmniVul, a comprehensive multi-turn benchmark for LLM-based vulnerability assessment. OmniVul comprises 2,000 CVEs with question–answer pairs spanning 23 attributes, including detection, code localization, root cause analysis, and patch suggestion. We employ an automated workflow to aggregate multi-source data via Retrieval-Augmented Generation (RAG), ensuring quality through LLM-as-a-Judge filtering and conformal prediction calibrated by human expert annotations. An evaluation of five state-of-the-art LLMs on OmniVul reveals distinct performance gaps, with top-1 accuracy remaining below 50% on average for vulnerable code detection and CVE identification. Our evaluation also demonstrates that current models lack critical reasoning capabilities for reliable vulnerability assessment. These results highlight the importance of OmniVul for advancing research in evaluating and fine-tuning LLMs for vulnerability assessment.
Vishnu Teja Kandalam, Viet Duong, Xiaochang Li et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
