Meta-evaluation of text generation metrics and LLM-judges is primarily conducted using metric-human correlations. However, prior work has shown that correlation is not an appropriate estimate of a metric's evaluation capability and can be unstable across evaluation settings, such as different datasets or languages. The...
We introduce Type-6 logic, a variant of dynamic epistemic logic augmented with two operators (uncertainty and recurrence), designed to model the inferential dynamics of contemporary large language model (LLM) chain-of-thought (CoT) reasoning. Type-6 accounts for common LLM reasoning pathologies such as unlicensed revis...
This work critically examines the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges, and finds strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods.
Anubhav Jangra, Bahareh Sarrafzadeh, Adrian de Wynter et al.· arXiv.org· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.