Skip to content

Energy-Based Transformers as Predictors of Reading Difficulty

Jun 2026 · arXiv.org · Vol abs/2606.23382 · 1 citation
Computer Science

TL;DR

Evidence is found that energy may serve as a single unified predictor where multiple complementary measures have previously been required, suggesting that energy may serve as a single unified predictor where multiple complementary measures have previously been required.

Abstract

Transformer language models have become established tools for modeling human sentence processing, with measures such as surprisal and attention entropy serving as effective predictors of reading difficulty that together capture complementary aspects of processing load. Here, we explore a related class of transformer models: energy-based transformers, which provide a principled formal link to associative memory models, bringing processing research into direct contact with the broader literature on Hopfield networks and dense associative memory. To our knowledge, this is the first exploration of an energy-based transformer measure in computational psycholinguistics. Across reading-time corpora (Natural Stories, UCL eye-tracking, UCL self-paced reading), the energy measure is a robust predictor of reading times, providing significant fit beyond surprisal and entropy in all three. In a controlled experiment on relative clause processing, energy at a single layer captures the well-known object/subject asymmetry. We find evidence that it subsumes effects attributable to both attention entropy and surprisal, suggesting that energy may serve as a single unified predictor where multiple complementary measures have previously been required.

View source

Similar papers

Preprint Aug 2026

Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?

A systematic comparison of multiple attention-based memory mechanisms across different model sizes and training corpora finds that constraints that are sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperforming distance-based constraints.

Lanni Bu, Xiulin Yang, Christian Clark et al. · 0 citations
Open access Aug 2026

Modeling Memory Effects in a Head-Final Language With Category Locality

Abstract Memory limitations have been assumed to be a major factor shaping human sentence comprehension, but characterizing memory-demanding structures in a broad-coverage and cross-linguistic manner has remained a challenge. A recent study suggested that such a general characterization can be obtained by assuming effi...

Shinnosuke Isono, Kohei Kajikawa, Yohei Oseki et al. · 2 citations
Open access Sep 2026

Neural tracking of surprisal and semantic distance in naturalistic movie viewing

Understanding speech requires listeners to integrate incoming input with prior linguistic and thematic knowledge to access meaning, a task greatly aided by prediction. Surprisal and related phenomena (e.g., next word prediction) tend to be associated with broad activation of language regions during listening. A major c...

Ryan M. O'Leary, Hailey C. Smith, Emily B. Myers et al. · 0 citations
Review Open access Sep 2026

Evaluating the compatibility of predictive coding as a model of linguistic predictability effects in reading

Emerging work suggests that the computational architecture of predictive coding (PC), widely studied in visual perception, may be an appropriate model for understanding predictability effects in real-time reading (i. e., the influence of a preceding sentence context on current word processing via putative anticipatory...

Allyson Copeland, B. Payne · 0 citations
#natural language process... Conference Open access Sep 2026

Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension

This work systematically vary the context window size of GPT-2 across four large-scale naturalistic English reading-time datasets and observes a U-shaped relationship, demonstrating that tracking long-range coreference relations is one important factor for the alignment between LM surprisal and human reading behavior.

Kohei Kajikawa, Lin Ai, Tatsuki Kuribayashi et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.