Nicolas Zucchet

Language models suffer from a curse of ambiguity

This work identifies a curse of ambiguity: in large language models, and more broadly in all neural networks that produce discrete probability distributions, the more ambiguous a next-token distribution is, the harder it is to learn accurately.