COA-Bench is presented, a small offline benchmark and reproducibility artifact for comparing COA generation policies through self-play, which represents COAs as typed action chains with conditional branches, assigns a synthetic COA quality score, and scores doctrinal coherence with an FM 3-0-inspired heuristic rubric.
Natan Vidra, Alina Kapanova, Arun Kanhai et al.· 0 citations
Multi-agent orchestration frameworks are moving from demos to production, yet benchmarks typically report task accuracy without diagnosing why a pipeline failed, where a cascade began, or which routing decision caused the breakdown. OrchestraBench evaluates failure, recovery, and decomposition through a controlled, see...
Yidian Chen, Ying Gu, Natan Vidra et al.· 2 citations
This work introduces an executable benchmark and a budget-aware meta-router that composes heterogeneous operations from raw task text that establishes a reproducible testbed and a bounded proof of concept, not evidence of live-LLM performance.
Natan Vidra, Alina Kapanova, Arun Kanhai et al.· 0 citations
Within low resource domains, results identify the model's parsing of information and subsequent reasoning as the source of reasoning failure, rather than corpus contents, rather than corpus contents.
This work presents MetaRoute-Bench, an open, inspectable framework for comparing meta-decision policies under a shared execution model, and releases task generation, policies, traces, tests, and analysis artifacts to support live-system validation.
Natan Vidra, Alina Kapanova, Arun Kanhai et al.· 0 citations
Security-adjusted reliability@k is proposed, which counts only rollouts that are both functionally correct and free of high-severity insecure patterns, which counts only rollouts that are both functionally correct and free of high-severity insecure patterns.
