Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
TL;DR
ProofRank is introduced, a benchmark curated from challenging competitions that measures several scalable proxies of proof quality and captures important aspects of mathematical proofs beyond correctness, and indicates the need for models that produce proofs which are not only correct but also valuable.
Abstract
Large language models (LLMs) have become capable mathematical problem-solvers, often producing correct proofs for challenging problems. However, correctness alone is not sufficient: mathematical proofs should also be clear, insightful, contain transferable knowledge, and be generally useful to the reader. While proof quality is subjective and depends on the reader and context, many of its components can be made concrete. In this work, we identify such components and introduce ProofRank, a benchmark curated from challenging competitions that measures several scalable proxies of proof quality: (i) conciseness - quantifying whether proofs avoid unnecessary steps; (ii) computational ease - the extent to which a proof relies on tedious calculations; (iii) cognitive simplicity - how accessible the used proof techniques are; (iv) diversity - how varied a model's proofs for a single problem are; and (v) adaptivity - how well a model can follow a specified technique. We find proof quality varies substantially across models along dimensions that correctness largely fails to capture. Every proof-level proxy can be steered through prompting, but often at a moderate correctness cost. ProofRank thus captures important aspects of mathematical proofs beyond correctness, and indicates the need for models that produce proofs which are not only correct but also valuable.
