Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions

• URL: http://arxiv.org/abs/2607.19378v1
• Date: Wed, 01 Jul 2026 02:39:18 GMT
• Title: Native Multi-Dimensional Subquadratic Operators via Input Dependent Long Convolutions
• Abstract summary: Subquadratic alternatives to attention require compromises when applied to multi-dimensional data.<n>We introduce textitHyenaND, a subquadratic, global, input-dependent operator that acts directly on the native geometry of convolutions.<n>Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines, while hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids.
• Score: 14.854950151689062
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Subquadratic alternatives to attention require compromises when applied to multi-dimensional data: standard convolutions lack global receptive fields and input dependency, while recurrent models require rasterizing data such as images, volumes, and partial differential equation (PDE) into an ad-hoc $1\rm D$ scan order that violates their spatial structure. We introduce \textit{HyenaND}, a subquadratic, global, input-dependent operator that acts directly on the native geometry of multidimensional data through convolutions with implicitly parametrized global, input-dependent multi-dimensional convolutional kernels. Our CUDA implementation, \texttt{nSubQ}, fuses the FFT-convolution path to turn HyenaND's $\mathcal{O}(L \log L)$ scaling into wall-clock speedups. Across long-context genomics, computer vision, medical imaging, and PDE modeling, pure HyenaND stacks match the accuracy of strong attention baselines, while hybrid configurations that interleave HyenaND and attention layers outperform both pure attention and strong recurrence-based hybrids.

Related papers

• Looped SSMs: Depth-Recurrence and Input Reshaping for Time Series Classification [50.994194925685434]
  We show that a looped SSM with $k$ parameters iterated $L$ times consistently closely matches or outperforms a standard SSM.<n>We also show that input reshaping is an equally neglected design axis.
  arXiv  Detail & Related papers  (2026-05-15T15:18:12Z)
• U-HNO: A U-shaped Hybrid Neural Operator with Sparse-Point Adaptive Routing for Non-stationary PDE Dynamics [7.23685260715089]
  We propose U-HNO, a U-shaped hybrid neural operator whose central design is Sparse-Point Adaptive Routing (SPAR)<n>SPAR is embedded in a hierarchical encoder-bottleneck-decoder backbone with skip connections so that the dual branches and the gate operate at every resolution.<n>U-HNO achieves state-of-the-art rollout accuracy on the majority of tasks in both relative L2 and H1 metrics, with the largest gains on problems dominated by sharp localized features.
  arXiv  Detail & Related papers  (2026-05-13T04:00:43Z)
• AE-ViT: Stable Long-Horizon Parametric Partial Differential Equations Modeling [0.0]
  We propose a joint model consisting of a convolutional encoder, a transformer operating on latent representations, and a decoder for reconstruction.<n>Our approach combines the efficiency of latent evolution with the fidelity of full-field models, outperforming DL-ROMs, latent transformers, and plain ViTs in multi-field prediction.
  arXiv  Detail & Related papers  (2026-04-07T21:19:45Z)
• Stochastic Dimension Implicit Functional Projections for Exact Integral Conservation in High-Dimensional PINNs [0.0]
  Enmath exact conservation laws as mass and energy, in neural partial differential equation (PDE) solvers is challenging in high dimensions.<n>For training, we introduce a-stochcal sampling from operator, the DS-I|timesDS (DS-I) to mitigate memory from such.<n>mathcalI|timesDS provides a scalable scalable approach for solving.<n>conservative high-dimensional PDEs.
  arXiv  Detail & Related papers  (2026-03-31T04:07:51Z)
• Spectral-Window Hybrid (SWH) [0.0]
  Scaling sequence modeling to extreme contexts requires balancing computational efficiency with representational expressivity.<n>We propose the textbfSpectral-Window Hybrid (SWH), an architecture that decouples sequence modeling into two textitparallel streams.<n>We demonstrate that SWH matches the perplexity of standard Transformers on short contexts while enabling efficient linear scaling to extended sequences.
  arXiv  Detail & Related papers  (2026-01-04T00:31:36Z)
• GSPN-2: Efficient Parallel Sequence Modeling [101.33780567131716]
  Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme.<n>GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications.
  arXiv  Detail & Related papers  (2025-11-28T07:26:45Z)
• Adaptive Mesh-Quantization for Neural PDE Solvers [51.26961483962011]
  Graph Neural Networks can handle the irregular meshes required for complex geometries and boundary conditions, but still apply uniform computational effort across all nodes.<n>We propose Adaptive Mesh Quantization: spatially adaptive quantization across mesh node, edge, and cluster features, dynamically adjusting the bit-width used by a quantized model.<n>We demonstrate our framework's effectiveness by integrating it with two state-of-the-art models, MP-PDE and GraphViT, to evaluate performance across multiple tasks.
  arXiv  Detail & Related papers  (2025-11-23T14:47:24Z)
• PiT: Progressive Diffusion Transformer [50.46345527963736]
  Diffusion Transformers (DiTs) achieve remarkable performance within image generation via the transformer architecture.<n>We find that DiTs do not rely as heavily on global information as previously believed.<n>We propose a series of Pseudo Progressive Diffusion Transformer (PiT)
  arXiv  Detail & Related papers  (2025-05-19T15:02:33Z)
• Parallel Sequence Modeling via Generalized Spatial Propagation Network [80.66202109995726]
  Generalized Spatial Propagation Network (GSPN) is a new attention mechanism for optimized vision tasks that inherently captures 2D spatial structures.<n>GSPN overcomes limitations by directly operating on spatially coherent image data and forming dense pairwise connections through a line-scan approach.<n>GSPN achieves superior spatial fidelity and state-of-the-art performance in vision tasks, including ImageNet classification, class-guided image generation, and text-to-image generation.
  arXiv  Detail & Related papers  (2025-01-21T18:56:19Z)
• Efficient High-Resolution Visual Representation Learning with State Space Model for Human Pose Estimation [60.80423207808076]
  Capturing long-range dependencies while preserving high-resolution visual representations is crucial for dense prediction tasks such as human pose estimation.<n>We propose the Dynamic Visual State Space (DVSS) block, which augments visual state space models with multi-scale convolutional operations.<n>We build HRVMamba, a novel model for efficient high-resolution representation learning.
  arXiv  Detail & Related papers  (2024-10-04T06:19:29Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• Polarized Self-Attention: Towards High-quality Pixel-wise Regression [19.2303932008785]
  This paper presents the Polarized Self-Attention(PSA) block that incorporates two critical designs towards high-quality pixel-wise regression. Experimental results show that PSA boosts standard baselines by $2-4$ points, and boosts state-of-the-arts by $1-2$ points on 2D pose estimation and semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-07-02T01:03:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.