This paper develops a measurement pipeline that recovers operational obligations, organizes their workflow context, maps testcases using both user prompts and initial resource states, and produces an auditable coverage report, which measures how completely a Skill’s test suite covers its operational test obligations.
Large language models are increasingly used to generate test scenarios, executable test suites, assertions, interaction sequences, and fuzzing infrastructure. These artifacts serve different purposes and rely on different sources of information about correct behavior. A test that increases implementation coverage, a te...
Yun-Hao Liang, Cheng-Guang Gan, Rui-Xuan Ying et al.· 0 citations
Executable Agent Skills combine natural-language instructions and scripts into reusable packages for LLM agents, and revising them requires fixing errors without breaking correct behavior. Existing benchmarks do not systematically distinguish documentation repair, script repair, and preservation when evaluating skill s...
Yuxuan Liu, Hao-Ran Li, Yu-Hao Zhang et al.· 2 citations
E2E-SWE is introduced, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end and finds substantial variation in end-to-end repository generation ability, providing strong model differentiation while leaving considerable headroom for future progress.
Results show that execution evidence can expose behavioral failures missed by artifact inspection and can guide Skill generation toward jointly verified functional and safety outcomes.
Zhi-Bo Zhang, Zheng-Mao Ouyang, Ling Shi et al.· 0 citations
Thinkingbox is introduced, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state, and Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing...
Zhuo-Chun Li, Youngmin Ko, Ali Keramati et al.· 3 citations
ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts, is presented.
Christopher Kevin, Narendran Raghavan, J. Puget et al.· 3 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
