Benchy: towards a universal language for task-oriented AI benchmarks
Benchy is a semantic language and execution engine for benchmarking AI programs. A benchmark is completely specified by a program, a scoring function, and a dataset, B=(P,S,D), and is separate from the AI-system taking it; a run binds the two, R=(B,AI). Benchmarks are authored as canonical YAML in which each semantic c...
Francis F. Daniel, Mauro Ibañez, Francis Perelman et al.
· 0 citations