A Boltzmann-machine-enhanced Transformer For DNA Sequence Classification

• URL: http://arxiv.org/abs/2603.26465v1
• Date: Fri, 27 Mar 2026 14:32:37 GMT
• Title: A Boltzmann-machine-enhanced Transformer For DNA Sequence Classification
• Abstract summary: We propose a Boltzmann-machine-enhanced Transformer for DNA sequence classification.<n>The model introduces structured binary gating variables to represent latent connections and constrains them with query-key connections.<n>During training, we jointly optimize classification and energy losses, encouraging the model to achieve accurate prediction.
• Score: 0.8711791966428426
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: DNA sequence classification requires not only high predictive accuracy but also the ability to uncover latent site interactions, combinatorial regulation, and epistasis-like higher-order dependencies. Although the standard Transformer provides strong global modeling capacity, its softmax attention is continuous, dense, and weakly constrained, making it better suited for information routing than explicit structure discovery. In this paper, we propose a Boltzmann-machine-enhanced Transformer for DNA sequence classification. Built on multi-head attention, the model introduces structured binary gating variables to represent latent query-key connections and constrains them with a Boltzmann-style energy function. Query-key similarity defines local bias terms, learnable pairwise interactions capture synergy and competition between edges, and latent hidden units model higher-order combinatorial dependencies. Since exact posterior inference over discrete gating graphs is intractable, we use mean-field variational inference to estimate edge activation probabilities and combine it with Gumbel-Softmax to progressively compress continuous probabilities into near-discrete gates while preserving end-to-end differentiability. During training, we jointly optimize classification and energy losses, encouraging the model to achieve accurate prediction while favoring low-energy, stable, and interpretable structures. We further derive the framework from the energy function and variational free energy to the mean-field fixed-point equations, Gumbel-Softmax relaxation, and the final joint objective. The proposed framework provides a unified view of integrating Boltzmann machines, differentiable discrete optimization, and Transformers for structured learning on biological sequences.

Related papers

• Hyper-Fold: Exploring the Expressive Limit of Sequence-Geometry Learning for Proteins via Hypergraph Modeling [55.36837305276515]
  We show that the complete bilinear operator over content-geometry outer products--the sufficient statistic of all second-order interactions--is the expressive ceiling.<n>We then introduce Hyper-Fold, a rank-K separable convolutional backbone approaching this ceiling at message-passing cost.
  arXiv  Detail & Related papers  (2026-08-29T11:43:25Z)
• Training-Free Universal Approximation by Prompting Random Transformers [48.556633593447465]
  We show that a softmax attention network with random, untrained weights can approximate any Hlder function on a compact manifold when steered by an appropriate soft prompt.<n>We further quantify the cost of prompting, exposing a tradeoff between the norm of the constructed soft prompt tokens, prompt length, and hidden dimension.
  arXiv  Detail & Related papers  (2026-08-10T12:57:22Z)
• Operator-Guided Invariance Learning for Continuous Reinforcement Learning [8.820825533010543]
  Reinforcement learning (RL) with continuous time and state/action spaces is often data-intensive and brittle under nuisance variability and shift.<n>We propose textbfVPSD-RL (Value-Preserving Structure Discovery for Reinforcement Learning)<n>It models continuous RL as a controlled diffusion with value-preserving mappings defined through Lie-group actions and associated pullback operators.<n>We show that a value-preserving structure exists exactly when pulling back the value function and pushing forward actions commute with the controlled generator and reward functional.
  arXiv  Detail & Related papers  (2026-05-07T16:18:37Z)
• Expanding functional protein sequence space using high entropy generative models [0.23090185577016445]
  Boltzmann Machines trained on evolutionary sequence data have emerged as a powerful paradigm for the data-driven design of artificial proteins.<n>We investigate the relationship between model architecture, specifically parameter density, and experimental performance.
  arXiv  Detail & Related papers  (2026-05-05T09:45:46Z)
• NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics [1.2026755069251662]
  We introduce the NeuroGame Transformer (NGT) to overcome standard attention mechanisms in transformers.<n> Token importance is quantified using two complementary game-theoretic concepts -- Shapley values for global, permutation-based attribution and Banzhaf indices for local, coalition-level influence.<n>The system's energy follows an Ising Hamiltonian, with attention weights emerging as marginal probabilities under the Gibbs distribution.
  arXiv  Detail & Related papers  (2026-03-19T11:14:30Z)
• Variational Bayesian Flow Network for Graph Generation [54.94088904387278]
  We propose Variational Bayesian Flow Network (VBFN) for graph generation.<n>VBFN performs variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions.<n>On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods.
  arXiv  Detail & Related papers  (2026-01-30T03:59:38Z)
• Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibrium [0.6820746164515952]
  We introduce the closed-loop prediction principle, which requires that models iteratively refine latent representations until reaching a self-consistent equilibrium.<n>We instantiate this principle as Equilibrium Transformers, which augment standard transformer layers with an Equilibrium Refinement Module.<n>Preliminary experiments on the binary parity task demonstrate +3.28% average improvement on challenging sequences, with gains reaching +8.07% where standard transformers approach random performance.
  arXiv  Detail & Related papers  (2025-11-26T20:02:59Z)
• PowerGrow: Feasible Co-Growth of Structures and Dynamics for Power Grid Synthesis [75.14189839277928]
  We present PowerGrow, a co-generative framework that significantly reduces computational overhead while maintaining operational validity.<n> Experiments across benchmark settings show that PowerGrow outperforms prior diffusion models in fidelity and diversity.<n>This demonstrates its ability to generate operationally valid and realistic power grid scenarios.
  arXiv  Detail & Related papers  (2025-08-29T01:47:27Z)
• BoltzNCE: Learning Likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation [1.2874523233023452]
  Efficient sampling from the Boltzmann distribution is a key challenge for modeling complex physical systems such as molecules.<n>We train an energy-based model (EBM) to approximate likelihoods using both noise contrastive estimation (NCE) and score matching.<n>Our approach also exhibits effective transfer learning, generalizing to new systems at inference time and achieving at least a $6times$ speedup over standard MD.
  arXiv  Detail & Related papers  (2025-07-01T15:18:28Z)
• Score-Based Model for Low-Rank Tensor Recovery [49.158601255093416]
  Low-rank tensor decompositions (TDs) provide an effective framework for multiway data analysis.<n>Traditional TD methods rely on predefined structural assumptions, such as CP or Tucker decompositions.<n>We propose a score-based model that eliminates the need for predefined structural or distributional assumptions.
  arXiv  Detail & Related papers  (2025-06-27T15:05:37Z)
• Scalable Equilibrium Sampling with Sequential Boltzmann Generators [60.00515282300297]
  We extend the Boltzmann generator framework with two key contributions.<n>The first is a highly efficient Transformer-based normalizing flow operating directly on all-atom Cartesian coordinates.<n>In particular, we perform inference-time scaling of flow samples using a continuous-time variant of sequential Monte Carlo.
  arXiv  Detail & Related papers  (2025-02-25T18:59:13Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation [85.5379146125199]
  Variational Auto-Encoder (VAE) has been widely adopted in text generation. We propose TRACE, a Transformer-based recurrent VAE structure.
  arXiv  Detail & Related papers  (2022-10-22T10:25:35Z)
• Sparse Communication via Mixed Distributions [29.170302047339174]
  We build theoretical foundations for "mixed random variables" Our framework suggests two strategies for representing and sampling mixed random variables. We experiment with both approaches on an emergent communication benchmark.
  arXiv  Detail & Related papers  (2021-08-05T14:49:03Z)
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers [42.93754828584075]
  We present a new Transformer architecture, Performer, based on Fast Attention Via Orthogonal Random features (FAVOR) Our mechanism scales linearly rather than quadratically in the number of tokens in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors. It provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence.
  arXiv  Detail & Related papers  (2020-06-05T17:09:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.