Boltzmann Attention: Learnable Ising Couplings for Cooperative Attention

• URL: http://arxiv.org/abs/2606.12478v1
• Date: Wed, 10 Jun 2026 06:58:49 GMT
• Title: Boltzmann Attention: Learnable Ising Couplings for Cooperative Attention
• Abstract summary: We propose Boltzmann attention, an energy-based generalization in which attention patterns are governed by an interacting Ising model.<n>Experiments on character-level language modeling and synthetic bracket matching show that Boltzmann attention consistently improves over standard softmax attention.
• Score: 0.21485350418225238
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention mechanisms are central to modern sequence models, yet standard attention computes relevance primarily through individual query--key similarities. Although softmax normalization introduces competition among positions, a standard attention layer does not explicitly parameterize learnable interactions between attention decisions. This limits its ability to directly model cooperative or antagonistic co-attention structure within the attention mechanism itself. We propose Boltzmann attention, an energy-based generalization in which attention patterns are governed by an interacting Ising model. The method augments the usual data-dependent local fields with learnable pairwise couplings, allowing the model to represent inter-position correlations beyond those captured by softmax or sigmoid attention. Experiments on character-level language modeling and synthetic bracket matching show that Boltzmann attention consistently improves over standard softmax attention within a standard Transformer architecture, with the advantage becoming more pronounced as sequence length increases. A four-way ablation confirms that the improvement arises from the learnable pairwise couplings. These results suggest that explicit inter-position interactions provide a principled enhancement for attention-based sequence modeling. Moreover, the Ising formulation opens a natural path toward quantum-computing-based sampling strategies: we demonstrate that diabatic quantum annealing provides a practical training method while maintaining competitive performance with exact Boltzmann computation.

Related papers

• Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention [10.26086309142631]
  GPT attention measures token compatibility through dot-product similarity.<n>It does not explicitly model whether strong token features should reinforce or suppress one another.<n>We introduce Q-Interference, a fully classical quantum-inspired attention mechanism for autoregressive language modeling.
  arXiv  Detail & Related papers  (2026-08-18T02:38:17Z)
• Krause Synchronization Transformers [63.8469912831803]
  Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer.<n>We introduce Krause Attention, a principled attention mechanism inspired by bounded-confidence consensus dynamics.
  arXiv  Detail & Related papers  (2026-02-12T03:47:53Z)
• Nexus: Higher-Order Attention Mechanisms in Transformers [82.07756094886552]
  Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies.<n>In this paper, we propose the Nexus, a novel architecture designed to enhance representational power through a recursion framework.<n>We provide theoretical analysis demonstrating that our method breaks the linear bottleneck of standard attention. Empirically, Nexus outperforms standard Transformers on multiple benchmarks.
  arXiv  Detail & Related papers  (2025-12-03T02:25:38Z)
• Deconstructing Attention: Investigating Design Principles for Effective Language Modeling [37.92951508140559]
  Transformer language models are widely credited with their dot-product attention mechanism.<n>This work systematically deconstructs attention by designing controlled variants that relax these principles.<n>Surprisingly, even variants that fail in isolation can achieve robust performance when interleaved with standard attention.
  arXiv  Detail & Related papers  (2025-10-13T16:42:14Z)
• Dynamic Relational Priming Improves Transformer in Multivariate Time Series [0.0]
  We propose attention with dynamic relational priming (prime attention)<n>We show that prime attention consistently outperforms standard attention across benchmarks.<n>We also find that prime attention achieves comparable or superior performance using up to 40% less sequence length compared to standard attention.
  arXiv  Detail & Related papers  (2025-09-15T17:56:15Z)
• Mitigating Attention Hacking in Preference-Based Reward Modeling via Interaction Distillation [62.14692332209628]
  "Interaction Distillation" is a novel training framework for more adequate preference modeling through attention-level optimization.<n>It provides more stable and generalizable reward signals compared to state-of-the-art RM optimization methods.
  arXiv  Detail & Related papers  (2025-08-04T17:06:23Z)
• Transformers Learn Faster with Semantic Focus [57.97235825738412]
  We study sparse transformers in terms of learnability and generalization.<n>We find that input-dependent sparse attention models appear to converge faster and generalize better than standard attention models.
  arXiv  Detail & Related papers  (2025-06-17T01:19:28Z)
• Revisiting Attention Weights as Explanations from an Information Theoretic Perspective [4.499369811647602]
  We show that attention mechanisms have the potential to function as a shortcut to model explanations when they are carefully combined with other model elements. Our findings indicate that attention mechanisms do have the potential to function as a shortcut to model explanations when they are carefully combined with other model elements.
  arXiv  Detail & Related papers  (2022-10-31T12:53:20Z)
• Learning Sequence Representations by Non-local Recurrent Neural Memory [61.65105481899744]
  We propose a Non-local Recurrent Neural Memory (NRNM) for supervised sequence representation learning. Our model is able to capture long-range dependencies and latent high-level features can be distilled by our model. Our model compares favorably against other state-of-the-art methods specifically designed for each of these sequence applications.
  arXiv  Detail & Related papers  (2022-07-20T07:26:15Z)
• An Empirical Study: Extensive Deep Temporal Point Process [61.14164208094238]
  We first review recent research emphasis and difficulties in modeling asynchronous event sequences with deep temporal point process.<n>We propose a Granger causality discovery framework for exploiting the relations among multi-types of events.
  arXiv  Detail & Related papers  (2021-10-19T10:15:00Z)
• Bayesian Attention Belief Networks [59.183311769616466]
  Attention-based neural networks have achieved state-of-the-art results on a wide range of tasks. This paper introduces Bayesian attention belief networks, which construct a decoder network by modeling unnormalized attention weights. We show that our method outperforms deterministic attention and state-of-the-art attention in accuracy, uncertainty estimation, generalization across domains, and adversarial attacks.
  arXiv  Detail & Related papers  (2021-06-09T17:46:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.