Interdomain Attention: Beyond Token-Level Key-Value Memory

• URL: http://arxiv.org/abs/2605.24330v1
• Date: Sat, 23 May 2026 01:18:00 GMT
• Title: Interdomain Attention: Beyond Token-Level Key-Value Memory
• Abstract summary: We propose Interdomain Attention, which integrates an SSM into an attention module through kernel methods.<n>An attention kernel is approximated by a finite feature map, the resulting key features and values are projected onto a shared set of basis functions.<n>Each query attends to the compressed coefficients through its own feature map, recovering query-conditioned attention over a fixed-size state.
• Score: 15.708551010135407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers and deep state space models (SSMs) sit at opposite ends of a basic design choice: attention routes each query through a growing key-value (KV) cache by content-based matching at quadratic cost, while deep SSMs compress context into a fixed-size recurrent state that is not directly addressed by query-key matching. We propose Interdomain Attention, which integrates an SSM into an attention module through kernel methods: an attention kernel is approximated by a finite feature map, the resulting key features and values are projected onto a shared set of basis functions maintained by a single SSM recurrence, and each query attends to the compressed coefficients through its own feature map, recovering query-conditioned attention over a fixed-size state. The scalable layer is a learned relaxation of this derivation, and we validate its components through ablations. In a 125M to 1.3B autoregressive language-modeling study on FineWeb-Edu at matched recurrent-state budget, Interdomain Attention improves on an SSM token mixer at every scale, surpasses a same-recipe softmax baseline at 1.3B on validation perplexity and on the eight-task commonsense suite, and inherits the length-flat behavior of its fixed-state core out to 3.5x the training context. Ablations indicate that the query-conditioned projection is the main source of the gain.

Related papers

• Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling [61.28599393707943]
  Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generation timesteps.<n>We propose a task-shared core scorer and progress-conditioned generation expansions, optimized through generation decomposition and cross-task core alignment.<n> Experiments on two representative UMM architectures demonstrate consistent quality--efficiency improvements across both tasks, retaining 98.03% of dense understanding performance with a 1.93times$ end-to-end inference speedup.
  arXiv  Detail & Related papers  (2026-08-29T14:27:55Z)
• RoleMix: Unifying Sequential and Non-Sequential Features via Semantic Tokenization for Post-Click Conversion Rate Prediction [1.6050320174037502]
  RoleMix represents sequential and non-sequential evidence through a shared, role-preserving token interface.<n>On the large-scale KDD Cup 2026 Tencent UniRec Challenge, RoleMix achieves 83.648% online AUC, outperforming the official industrial baseline by 1.953%.
  arXiv  Detail & Related papers  (2026-07-18T15:44:48Z)
• Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention [0.48342038441006796]
  We study a sparse cache that allocates a slot only when an input is novel, so its size tracks the number of distinct items rather than the number of tokens.<n>We develop it in two forms, a static cache with a fixed concentration and a surprise-adaptive variant whose concentration follows the recent novelty rate.
  arXiv  Detail & Related papers  (2026-07-10T18:28:21Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference [38.75965915663233]
  We propose StructKV, a structure-aware KV cache compression framework.<n>First, Global In-Degree Centrality aggregates attention patterns across the network depth to identify global information hubs.<n>Second, Dynamic Pivot Detection utilizes information-theoretic metrics to adaptively locate the optimal layer for compression.
  arXiv  Detail & Related papers  (2026-04-08T07:10:35Z)
• BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation [12.449732626554967]
  We present BALD-SAM, a principled framework adapting Bayesian Active Learning by Disagreement to spatial prompt selection.<n>BALD-SAM demonstrates strong cross-domain performance, ranking first or second on 14 of 16 benchmarks.<n>We validate these gains through a comprehensive ablation suite covering 3 SAM backbones and 35 Laplace posterior configurations.
  arXiv  Detail & Related papers  (2026-03-11T14:34:55Z)
• LaSSM: Efficient Semantic-Spatial Query Decoding via Local Aggregation and State Space Models for 3D Instance Segmentation [21.566771922153027]
  We introduce LaSSM, which prioritizes simplicity and efficiency while maintaining competitive performance.<n>We also present a coordinate-guided state space model (SSM) decoder that progressively refines queries.<n>LaSSM ranks first place on the latest ScanNet++ V2 leaderboard, outperforming the previous best method by 2.5% mAP with only 1/3 FLOPs.
  arXiv  Detail & Related papers  (2026-02-11T16:34:12Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models [44.28116882776357]
  We present textbfPunctuation-aware textbfHybrid textbfSparse textbfAttention textbf(PHSA), a trainable sparse attention framework that leverages punctuation tokens as semantic boundary anchors.<n>Specifically, we design a dual-branch aggregation mechanism that fuses global semantic representations with punctuation-enhanced boundary features, preserving the core semantic structure while introducing almost no additional computational overhead.
  arXiv  Detail & Related papers  (2026-01-06T08:47:16Z)
• Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing [55.291219073365546]
  Open-Vocabulary Remote Sensing Image (OVRSIS) is an emerging task that adapts Open-Vocabulary (OVS) to the remote sensing (RS) domain.<n>textbfRSKT-Seg is a novel open-vocabulary segmentation framework tailored for remote sensing.<n> RSKT-Seg consistently outperforms strong OVS baselines by +3.8 mIoU and +5.9 mACC, while achieving 2x faster inference through efficient aggregation.
  arXiv  Detail & Related papers  (2025-09-15T15:24:49Z)
• Core Context Aware Transformers for Long Context Language Modeling [50.774702091154204]
  We propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-context modeling.<n>Our method automatically focuses and strengthens core context while diminishing redundancy during the learning process.<n>Our method is able to replace the self-attention module in existing Large Language Models with minimal fine-tuning cost.
  arXiv  Detail & Related papers  (2024-12-17T01:54:08Z)
• StyDeSty: Min-Max Stylization and Destylization for Single Domain Generalization [85.18995948334592]
  Single domain generalization (single DG) aims at learning a robust model generalizable to unseen domains from only one training domain. State-of-the-art approaches have mostly relied on data augmentations, such as adversarial perturbation and style enhancement, to synthesize new data. We propose emphStyDeSty, which explicitly accounts for the alignment of the source and pseudo domains in the process of data augmentation.
  arXiv  Detail & Related papers  (2024-06-01T02:41:34Z)
• RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling [0.0]
  Relative Multi-Head Attention (RMHA) has superior accuracy, boasting a score of 0.572 in comparison to alternative attention modules. RMHA emerges as the most frugal, demonstrating an average consumption of 2.98 GB, surpassing RMHA which necessitates 3.5 GB.
  arXiv  Detail & Related papers  (2023-08-07T09:24:24Z)
• Unveiling the Potential of Structure-Preserving for Weakly Supervised Object Localization [71.79436685992128]
  We propose a two-stage approach, termed structure-preserving activation (SPA), towards fully leveraging the structure information incorporated in convolutional features for WSOL. In the first stage, a restricted activation module (RAM) is designed to alleviate the structure-missing issue caused by the classification network. In the second stage, we propose a post-process approach, termed self-correlation map generating (SCG) module to obtain structure-preserving localization maps.
  arXiv  Detail & Related papers  (2021-03-08T03:04:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.