NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics

• URL: http://arxiv.org/abs/2603.18761v1
• Date: Thu, 19 Mar 2026 11:14:30 GMT
• Title: NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics
• Abstract summary: We introduce the NeuroGame Transformer (NGT) to overcome standard attention mechanisms in transformers.<n> Token importance is quantified using two complementary game-theoretic concepts -- Shapley values for global, permutation-based attribution and Banzhaf indices for local, coalition-level influence.<n>The system's energy follows an Ising Hamiltonian, with attention weights emerging as marginal probabilities under the Gibbs distribution.
• Score: 1.2026755069251662
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard attention mechanisms in transformers are limited by their pairwise formulation, which hinders the modeling of higher-order dependencies among tokens. We introduce the NeuroGame Transformer (NGT) to overcome this by reconceptualizing attention through a dual perspective: tokens are treated simultaneously as players in a cooperative game and as interacting spins in a statistical physics system. Token importance is quantified using two complementary game-theoretic concepts -- Shapley values for global, permutation-based attribution and Banzhaf indices for local, coalition-level influence. These are combined via a learnable gating parameter to form an external magnetic field, while pairwise interaction potentials capture synergistic relationships. The system's energy follows an Ising Hamiltonian, with attention weights emerging as marginal probabilities under the Gibbs distribution, efficiently computed via mean-field equations. To ensure scalability despite the exponential coalition space, we develop importance-weighted Monte Carlo estimators with Gibbs-distributed weights. This approach avoids explicit exponential factors, ensuring numerical stability for long sequences. We provide theoretical convergence guarantees and characterize the fairness-sensitivity trade-off governed by the interpolation parameter. Experimental results demonstrate that the NeuroGame Transformer achieves strong performance across SNLI, and MNLI-matched, outperforming some major efficient transformer baselines. On SNLI, it attains a test accuracy of 86.4\% (with a peak validation accuracy of 86.6\%), surpassing ALBERT-Base and remaining highly competitive with RoBERTa-Base. Code is available at https://github.com/dbouchaffra/NeuroGame-Transformer.

Related papers

• Multipole splats for optimized and inverted effective potentials [0.4369550829556578]
  We show improvements to three canonical DFT failure modes.<n>By comparing approximate and near-exact inverted effective correlation potentials, we isolate the spatial signatures of self-interaction, delocalization, and static correlation errors.<n>We also compute exact-exchange potentials for larger $$-conjugated systems, scaling capacity.
  arXiv  Detail & Related papers  (2026-09-08T18:00:02Z)
• Parametric Open Source Games [78.29352305480285]
  We introduce parametric open-source games, a continuous analogue of program equilibria.<n>We derive an exact coupling threshold at which selfish gradient ascent in symmetric $2times2$ games switches from defection toward cooperation.<n>We extend the framework to a neural semantics class whose first-order cooperation condition is governed by the ratio of cross-player to self-player sensitivity.
  arXiv  Detail & Related papers  (2026-06-25T14:14:24Z)
• Elastic Spiking Transformers for Efficient Gesture Understanding [6.52225846892392]
  Spiking Neural Networks (SNNs) offer energy-efficient processing of event-based sensor data for healthcare applications.<n>We introduce the Elastic Spiking Transformer, a runtime-adaptive architecture that brings elasticity into the spiking paradigm.<n>We evaluate the approach on CIFAR10/100, CIFAR10-DVS, and the EHWGesture clinical gesture understanding dataset.
  arXiv  Detail & Related papers  (2026-05-04T12:35:52Z)
• A Boltzmann-machine-enhanced Transformer For DNA Sequence Classification [0.8711791966428426]
  We propose a Boltzmann-machine-enhanced Transformer for DNA sequence classification.<n>The model introduces structured binary gating variables to represent latent connections and constrains them with query-key connections.<n>During training, we jointly optimize classification and energy losses, encouraging the model to achieve accurate prediction.
  arXiv  Detail & Related papers  (2026-03-27T14:32:37Z)
• Neural Network Quantum Field Theory from Transformer Architectures [0.0]
  We propose a neural-network construction of Euclidean scalar quantum field theories from transformer attention heads.<n>For a single attention head, shared random softmax weights couple different width coordinates and induce non-Gaussian field statistics.<n>We show that summing many independent heads with standard $1/N_h$ normalization suppresses connected non-Gaussian correlators as $1/N_h$, yielding a Gaussian NN-QFT in the large-head limit.
  arXiv  Detail & Related papers  (2026-02-10T19:02:03Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Unregularized Linear Convergence in Zero-Sum Game from Preference Feedback [50.89125374999765]
  We provide the first convergence guarantee for Optimistic Multiplicative Weights Update ($mathtOMWU$) in NLHF.<n>Our analysis identifies a novel marginal convergence behavior, where the probability of rarely played actions grows exponentially from exponentially small values.
  arXiv  Detail & Related papers  (2025-12-31T12:08:29Z)
• Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibrium [0.6820746164515952]
  We introduce the closed-loop prediction principle, which requires that models iteratively refine latent representations until reaching a self-consistent equilibrium.<n>We instantiate this principle as Equilibrium Transformers, which augment standard transformer layers with an Equilibrium Refinement Module.<n>Preliminary experiments on the binary parity task demonstrate +3.28% average improvement on challenging sequences, with gains reaching +8.07% where standard transformers approach random performance.
  arXiv  Detail & Related papers  (2025-11-26T20:02:59Z)
• Spiking Transformer:Introducing Accurate Addition-Only Spiking Self-Attention for Transformer [15.93436166506258]
  Spiking Neural Networks have emerged as a promising energy-efficient alternative to traditional Artificial Neural Networks.<n>This paper introduces Accurate Addition-Only Spiking Self-Attention (A$2$OS$2$A)
  arXiv  Detail & Related papers  (2025-02-28T22:23:29Z)
• Kolmogorov-Arnold Transformer [72.88137795439407]
  We introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces layers with Kolmogorov-Arnold Network (KAN) layers. We identify three key challenges: (C1) Base function, (C2) Inefficiency, and (C3) Weight. With these designs, KAT outperforms traditional-based transformers.
  arXiv  Detail & Related papers  (2024-09-16T17:54:51Z)
• Variance-Reducing Couplings for Random Features [57.73648780299374]
  Random features (RFs) are a popular technique to scale up kernel methods in machine learning. We find couplings to improve RFs defined on both Euclidean and discrete input spaces. We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm.
  arXiv  Detail & Related papers  (2024-05-26T12:25:09Z)
• Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers [14.59741397670484]
  We consider a deep multi-head self-attention network, that is closely related to Transformers yet analytically tractable.<n>We develop a statistical mechanics theory of Bayesian learning in this model.<n> Experiments confirm our findings on both synthetic and real-world sequence classification tasks.
  arXiv  Detail & Related papers  (2024-05-24T20:34:18Z)
• Gibbs-Duhem-Informed Neural Networks for Binary Activity Coefficient Prediction [45.84205238554709]
  We propose Gibbs-Duhem-informed neural networks for the prediction of binary activity coefficients at varying compositions. We include the Gibbs-Duhem equation explicitly in the loss function for training neural networks.
  arXiv  Detail & Related papers  (2023-05-31T07:36:45Z)
• Better Regularization for Sequential Decision Spaces: Fast Convergence Rates for Nash, Correlated, and Team Equilibria [121.36609493711292]
  We study the application of iterative first-order methods to the problem of computing equilibria of large-scale two-player extensive-form games. By instantiating first-order methods with our regularizers, we develop the first accelerated first-order methods for computing correlated equilibra and ex-ante coordinated team equilibria.
  arXiv  Detail & Related papers  (2021-05-27T06:10:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.