Variational-Ising-Attention (VIA):TailoredAttentionMattersfor Science

• URL: http://arxiv.org/abs/2607.23634v1
• Date: Sun, 26 Jul 2026 12:44:36 GMT
• Title: Variational-Ising-Attention (VIA):TailoredAttentionMattersfor Science
• Abstract summary: We propose Variational-Ising-Attention, which augments softmax normalization with an interacting Ising model.<n>Experiments show that VIA consistently and substantially outperforms standard softmax attention.
• Score: 3.744103474408833
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Attention enables context modeling via query-key scoring with softmax normalization. Driven by industrial long-context demands, mainstream research has converged toward sparsity and efficiency--yet softmax's independence assumption persists. For scientific tasks unburdened by long-token constraints, however, richer structured coupling may often be essential, making tailored attention both viable and more appropriate. To this end, we propose Variational-Ising-Attention (VIA), which augments softmax normalization with an interacting Ising model; attention patterns emerge from learnable pairwise couplings via variational mean-field inference, redefining attention from a ranking over isolated items to a collective state over interacting entities. We instantiate VIA on retrosynthesis reaction center prediction, a task inherently governed by cooperative bond-breaking constraints. Comprehensive experiments across model variants, coupled with mechanistic analyses, demonstrate that VIA consistently and substantially outperforms standard softmax attention. More broadly, our findings suggest that for scientific problems, the optimal solution is not general-purpose efficiency, but appropriately tailored attention aligned with intrinsic domain structure. This work provides a theoretically grounded and empirically validated instantiation of this paradigm.

Related papers

• CDFM: Towards a General-Purpose Causal Discovery Foundation Model [62.32138962508216]
  Causal discovery is the process of recovering underlying causal structures from observational data.<n>We formulate the Causal Discovery Foundation Model (CDFM) as a unified, general-purpose framework for zero-shot structural inference.<n>CDFM consistently outperforms traditional algorithms, driving a paradigm shift toward a general-purpose causal discovery foundation model.
  arXiv  Detail & Related papers  (2026-07-13T12:58:28Z)
• FedCausal-Dyn: A Causal-Dynamic Paradigm for Federated Learning under Dynamic Feature Drift [3.1354593336919883]
  We propose textbfFedCausal-Dyn, a novel federated learning framework built on a causal-dynamic paradigm.<n>Key innovation is textitcausal-domain feature separation, which disentangles domain-invariant causal features.<n>Experiments on three federated domain generalization benchmarks demonstrate that FedCausal-Dyn consistently achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-06-20T04:02:49Z)
• Improving Adversarial Robustness of Attribution via Implicit Regularization [9.78455276849371]
  adversarial robustness of attributions is a fundamental requirement for reliable explainability in deep learning.<n>We show that robustness can arise implicitly from the learning dynamics of standard gradient descent.<n>Our results highlight learning dynamics as a principled and practical mechanism for robust explainability.
  arXiv  Detail & Related papers  (2026-05-28T14:19:39Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuning [20.73438231053258]
  This paper proposes a comprehensive framework adapted to the multi-modal concept drift.<n>It integrates counterfactual reasoning with domain knowledge to execute controlled perturbations across thinking and perception.<n>It achieves superior performance in reasoning coherence, decision-making precision, and inherent robustness against extreme interference.
  arXiv  Detail & Related papers  (2026-04-17T05:24:04Z)
• OMNIFLOW: A Physics-Grounded Multimodal Agent for Generalized Scientific Reasoning [51.33849811496781]
  Large Language Models (LLMs) have demonstrated exceptional logical reasoning capabilities but frequently struggle with the continuous dynamics by Partial Differential Equations (PDEs)<n>We propose OMNIFLOW, a multimodal-symbolic architecture designed to ground frozen LLMs in fundamental physical laws without requiring domain-specific parameter updates.<n>We evaluate it on a benchmark spanning microscopic turbulence, theoretical Navier-Stokes, and macroscopic global weather forecasting.
  arXiv  Detail & Related papers  (2026-03-16T18:29:01Z)
• Is Softmax Loss All You Need? A Principled Analysis of Softmax-family Loss [91.61796429377041]
  The Softmax loss is one of the most widely employed surrogate objectives for classification and ranking tasks.<n>We investigate whether different surrogates achieve consistency with classification and ranking metrics, and analyze their gradient dynamics to reveal distinct convergence behaviors.<n>Our results establish a principled foundation and offer practical guidance for loss selections in large-class machine learning applications.
  arXiv  Detail & Related papers  (2026-01-30T09:24:52Z)
• A Generalized Adaptive Joint Learning Framework for High-Dimensional Time-Varying Models [0.8594140167290097]
  This article introduces Adaptive Joint Learning (AJL), a regularization framework designed to simultaneously perform functional variable selection and structural changepoint detection.<n>The analysis uncovers synchronized phase transitions in disease progression and identifies a parsimonious set of time-varying prognostic markers.
  arXiv  Detail & Related papers  (2026-01-08T02:07:49Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• Adaptive Termination for Multi-round Parallel Reasoning: An Universal Semantic Entropy-Guided Framework [12.361554676966552]
  Recent advances in large language models (LLMs) have accelerated progress toward artificial general intelligence.<n>We aim to design a flexible test-time collaborative inference framework that exploits the complementary strengths of both sequential and parallel reasoning paradigms.
  arXiv  Detail & Related papers  (2025-07-09T13:28:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.