PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics
A novel type of benchmark evaluating the inductive reasoning capabilities of LLMs that is inspired by the forward reconstruction task from historical linguistics but is formulated in an extremely simple, general way (in the form of Programming by Examples).
Atharva Naik, Prakam, Yash Mathur et al.
· Annual Meeting of the Associ... · 2 citations