Energy-Based Transformers as Predictors of Reading Difficulty

• URL: http://arxiv.org/abs/2606.23382v1
• Date: Mon, 22 Jun 2026 14:11:29 GMT
• Title: Energy-Based Transformers as Predictors of Reading Difficulty
• Abstract summary: We explore a related class of transformer models: energy-based transformers.<n>This is the first exploration of an energy-based transformer measure in computational psycholinguistics.<n>We find evidence that it subsumes effects attributable to both attention entropy and surprisal, suggesting that energy may serve as a single unified predictor.
• Score: 1.0597393315808434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer language models have become established tools for modeling human sentence processing, with measures such as surprisal and attention entropy serving as effective predictors of reading difficulty that together capture complementary aspects of processing load. Here, we explore a related class of transformer models: energy-based transformers, which provide a principled formal link to associative memory models, bringing processing research into direct contact with the broader literature on Hopfield networks and dense associative memory. To our knowledge, this is the first exploration of an energy-based transformer measure in computational psycholinguistics. Across reading-time corpora (Natural Stories, UCL eye-tracking, UCL self-paced reading), the energy measure is a robust predictor of reading times, providing significant fit beyond surprisal in all three. In a controlled experiment on relative clause processing, energy at a single layer captures the well-known object/subject asymmetry. We find evidence that it subsumes effects attributable to both attention entropy and surprisal, suggesting that energy may serve as a single unified predictor where multiple complementary measures have previously been required.

Related papers

• Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling [20.11728693585654]
  We show that frozen transformers can simulate iterative samplers from in-context samples.<n>We first show that transformers can realize closed-form and smoothed closed-form diffusion samplers.<n>We then prove that transformers can approximate an energy-based sampler, constructing the same U-shape energy across the layers.
  arXiv  Detail & Related papers  (2026-09-08T16:25:11Z)
• From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs [6.041312734387813]
  This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating large language model (LLM) inference energy on NVIDIA H100-class accelerators without direct measurement.<n>The proposed estimator combines parameter-scaled transformer FLOP accounting, memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement.<n>The methodology further decomposes total energy into compute, parameter-access, key-value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to
  arXiv  Detail & Related papers  (2026-07-29T07:50:55Z)
• Fine-Tuning Dynamics of In-Context Factual Recall in Transformers [63.1821972855273]
  In-context learning -- performing tasks based on examples given in the prompt -- has emerged in large language models.<n>We study how incontext learning leverages factual knowledge recall.<n>We analyze the supervised fine-tuning dynamics of a one-layer transformer on IC-recall data.
  arXiv  Detail & Related papers  (2026-05-26T23:47:26Z)
• Revisiting Transformer Layer Parameterization Through Causal Energy Minimization [19.719105230471282]
  We introduce Causal Energy Minimization (CEM), a framework that recasts Transformer layers as optimization steps on conditional energy functions.<n>We show that CEM connects Transformer architectures to energy-based models and motivate further exploration of energy-guided layer designs.
  arXiv  Detail & Related papers  (2026-05-08T11:02:11Z)
• Image Recognition with Online Lightweight Vision Transformer: A Survey [53.005965123414576]
  This paper surveys various online strategies for generating lightweight vision transformers for image recognition.<n>We evaluate the relevant exploration for each topic on the ImageNet-1K benchmark.<n>We propose future research directions and potential challenges in the lightweighting of vision transformers.
  arXiv  Detail & Related papers  (2025-05-06T02:07:54Z)
• Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference [0.30104001512119216]
  Building models with fast and energy-efficient inference is imperative to enable a variety of transformer-based applications. We build on an approach for learning LUT networks directly via an Extended Finite Difference method. This allows for a computational and energy-efficient inference solution for transformer-based models.
  arXiv  Detail & Related papers  (2024-11-04T05:38:56Z)
• Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory [11.3128832831327]
  Increasing the size of a Transformer does not always lead to enhanced performance.<n>We present a theoretical framework that sheds light on the memorization during pre-training of transformer-based language models.
  arXiv  Detail & Related papers  (2024-05-14T15:48:36Z)
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks [60.38369406877899]
  Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. Our survey encompasses the identification of the top five application domains for transformer-based models.
  arXiv  Detail & Related papers  (2023-06-11T23:13:51Z)
• Transformers from an Optimization Perspective [24.78739299952529]
  We study the problem of finding an energy function underlying the Transformer model. By finding such a function, we can reinterpret Transformers as the unfolding of an interpretable optimization process. This work contributes to our intuition and understanding of Transformers, while potentially laying the ground-work for new model designs.
  arXiv  Detail & Related papers  (2022-05-27T10:45:15Z)
• Attention Mechanism with Energy-Friendly Operations [61.58748425876866]
  We rethink attention mechanism from the energy consumption aspects. We build a novel attention model by replacing multiplications with either selective operations or additions. Empirical results on three machine translation tasks demonstrate that the proposed model achieves competitable accuracy.
  arXiv  Detail & Related papers  (2022-04-28T08:50:09Z)
• Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction [51.80191416661064]
  We propose a novel vision transformer with latent variables following an informative energy-based prior for salient object detection. Both the vision transformer network and the energy-based prior model are jointly trained via Markov chain Monte Carlo-based maximum likelihood estimation. With the generative vision transformer, we can easily obtain a pixel-wise uncertainty map from an image, which indicates the model confidence in predicting saliency from the image.
  arXiv  Detail & Related papers  (2021-12-27T06:04:33Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.