Adaptive Anisotropic Attention for Axis-Structured Signals

• URL: http://arxiv.org/abs/2609.08788v2
• Date: Tue, 15 Sep 2026 05:15:15 GMT
• Title: Adaptive Anisotropic Attention for Axis-Structured Signals
• Abstract summary: Self-attention treats all token pairs as equally plausible before learning, an interaction-isotropic prior that can be mismatched to structured signals.<n>We introduce Adaptive Anisotropic Attention (AAA), which splits attention into two paths: a temporal path and a spatial path.<n>We show that both paths are necessary and that the weighted sum beats a hard choice of one path; most of the benefit comes from the gate learning a different temporal/spatial balance at each layer of the network.
• Score: 5.616707402426108
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Dense self-attention treats all token pairs as equally plausible before learning, an interaction-isotropic prior that can be mismatched to structured signals. For structured, low signal-to-noise ratio (SNR) signals such as EEG, dependencies are organized along the electrode and time axes, and this uniform prior exposes each token to many irrelevant interactions. We introduce Adaptive Anisotropic Attention (AAA), which splits attention into two paths: a temporal path, where each token attends to the tokens of its own electrode across time, and a spatial path, where it attends to the tokens of the other electrodes at the same time step. A small gate predicts, for every token, a convex combination of the two path outputs: two non-negative weights that sum to one. On six EEG downstream tasks, the resulting model, AXON (AXis-factorized Operator Network), improves mean balanced accuracy over a dense baseline under both linear probing and full fine-tuning. We show that both paths (temporal and spatial) are necessary and that the weighted sum beats a hard choice of one path; most of the benefit comes from the gate learning a different temporal/spatial balance at each layer of the network. Controlled audio spectrogram experiments show that axis factorization transfers beyond EEG. These results suggest that aligning attention with the natural axes of structured signals provides a useful inductive bias.

Related papers

• From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers [53.505136935517925]
  Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood.<n>We prove that single-head attention (SHA) is exactly a connection propagation step with constant transport.<n>We find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory.
  arXiv  Detail & Related papers  (2026-07-12T09:44:54Z)
• What Does a Discrete Diffusion Model Learn? [71.03603607324338]
  discrete diffusion model learns: a denoiser, a score ratio, or a bridge plug-in predictor.<n>We start with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process.<n>Every identity is verified numerically, without approximation, on an exactly solvable model.
  arXiv  Detail & Related papers  (2026-07-06T17:56:11Z)
• Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention [0.0]
  We identify two complementary inductive biases that standard attention lacks.<n>Salience and locality are complementary inductive biases, each addressing a gap the other cannot fill alone.<n>This superadditivity, observed across two independent training runs, is the central empirical finding.
  arXiv  Detail & Related papers  (2026-05-25T22:04:31Z)
• SEED: Targeted Data Selection by Weighted Independent Set [76.68391670109433]
  We develop a robust and scalable data selection pipeline dubbed SEED.<n>SEED consistently outperforms state-of-the-art methods on instruction tuning, visual instruction tuning, and semantic segmentation.
  arXiv  Detail & Related papers  (2026-05-15T07:26:54Z)
• The Exponentially Weighted Signature [41.02458927230264]
  We introduce the Exponentially Weighted Signature (EWS)<n>We show that the EWS is the unique solution to a linear controlled differential equation on the tensor algebra.<n>We use this framework to empirically demonstrate the expressivity gap between the EWS and both the signature and EFM on two SDE-based regression tasks.
  arXiv  Detail & Related papers  (2026-03-19T17:51:20Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Missing-Data-Induced Phase Transitions in Spectral PLS for Multimodal Learning [2.9748898344267776]
  We study PLS-SVD under independent entry-wise missing-completely-at-random masking.<n>PLS-SVD exhibits a sharp BBP-type phase transition below a critical signal-to-noise threshold.<n> multimodal experiments corroborate the predicted phase diagram and recovery curves across aspect ratios, signal strengths, and missingness levels.
  arXiv  Detail & Related papers  (2026-01-29T05:46:44Z)
• Lightweight Transformer for EEG Classification via Balanced Signed Graph Algorithm Unrolling [22.934749236365178]
  EEG signals have inherent anti-correlations that are well modeled by negative edges in a finite graph.<n>We build lightweight and interpretable transformer-like neural nets by unrolling a spectral denoising algorithm for signals on a balanced signed graph.<n>Experiments show that our method achieves classification performance comparable to representative deep learning schemes.
  arXiv  Detail & Related papers  (2025-10-03T14:08:42Z)
• Biaxialformer: Leveraging Channel Independence and Inter-Channel Correlations in EEG Signal Decoding for Predicting Neurological Outcomes [1.7656489005289302]
  Accurate decoding of EEG signals requires modeling of both temporal dynamics within individual channels and spatial dependencies across channels.<n> Transformer-based models utilizing channel-independence (CI) strategies have demonstrated strong performance in various time series tasks.<n>We propose Biformer, characterized by a meticulously engineered two-stage attention-based framework.
  arXiv  Detail & Related papers  (2025-06-18T11:47:26Z)
• Graph Neural Networks for Edge Signals: Orientation Equivariance and Invariance [50.277959544420455]
  We develop EIGN, an architecture composed of novel direction-aware edge-level graph shift operators.<n>EIGN outperforms prior work in edge-level tasks, for example, improving in RMSE on flow simulation tasks by up to 23.5%.
  arXiv  Detail & Related papers  (2024-10-22T12:12:43Z)
• Adaptive Annealed Importance Sampling with Constant Rate Progress [68.8204255655161]
  Annealed Importance Sampling (AIS) synthesizes weighted samples from an intractable distribution. We propose the Constant Rate AIS algorithm and its efficient implementation for $alpha$-divergences.
  arXiv  Detail & Related papers  (2023-06-27T08:15:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.