A deterministic routing stress test over 20,000 skills shows the functional impact: skills with valid routing metadata are retrieved more reliably from startup descriptions than skills with routing defects, while AI-marked skills show more safety and portability problems.
Abstract
Under the current standard, Agent Skills are SKILL.md files that combine instructions with supporting resources, enabling Large Language Model (LLM) agents to reuse procedures beyond a single conversation. Yet many public skills appear to originate from a single task, repository, or conversation, even when they are shared as reusable components. We analyze this gap across 138,133 public SKILL.md files from 20,556 repositories using a two-tier defect taxonomy grounded in the official specification and best-practice guidance. We find that 91.8% of skills contain at least one detected defect, with stable estimates across lenient and strict thresholds (88.8-94.6%). The dominant failures are ordinary packaging problems rather than exotic attacks: weak routing metadata, bloated or non-actionable bodies, and poor resource organization. A deterministic routing stress test over 20,000 skills shows the functional impact: skills with valid routing metadata are retrieved more reliably from startup descriptions than skills with routing defects. Defect rates vary by platform and provenance: specification-aware skills contain fewer defects, while AI-marked skills show more safety and portability problems. Lightweight enforcement and repair experiments support a quality-assured generation workflow combining spec-aware prompting, lightweight linting, automated repair, and safety gating.
ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts, is presented.
Christopher Kevin, Narendran Raghavan, J. Puget et al.· 3 citations· ⚡1
These findings recast a matched Skill as a hypothesis about a particular Skill-project-model triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.
This work introduces SkillLogic, a framework for analyzing logical relations in skill files and constructing executable tests from them, and establishes logical-relation following as a distinct reliability challenge for skill-guided agents.
Xuan Chen, Chengpeng Wang, Lu Yan et al.· arXiv.org· 0 citations
SkillShift is presented, a constrained black-box framework for covert policy steering without explicit target command injection or task hijacking that combines semantically plausible policy edits with hierarchical validation, failure-guided optimization, and strategy compression to preserve effectiveness, output validi...
Jia-Rui Li, Jia-Hao Chen, Chun-Yi Zhou et al.· 0 citations
GitSkills is presented, a dataset of 3,797,117 $\mathrm{SKILL.md}$ files collected from 282,200 public repositories in July 2026, which retains every file occurrence with its repository, path, and content hash.
Giuseppe Destefanis, Daniel Graziotin, Matteo Vaccargiu et al.· 1 citation
This work designs a contrastive study that combines controlled quantitative experiments with paired trajectory analysis and consolidates observations into a taxonomy of three high-level categories and twelve skill-use modes, showing that skills work when noisy trajectories become procedural anchors that stabilize execu...
Zhi-Yuan Jiang, Fan Huang, Hanwen Xing et al.· 4 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.