RubiConv -- Efficient Boundary-Respecting Convolutions

• URL: http://arxiv.org/abs/2605.08451v1
• Date: Fri, 08 May 2026 20:17:12 GMT
• Title: RubiConv -- Efficient Boundary-Respecting Convolutions
• Abstract summary: We develop a novel algorithm for performing hardware-efficient, boundary-respecting convolutions on packed sequences.<n>RubiConv achieves significant speedups over both attention and standard FFT-based baselines.
• Score: 16.714478912341708
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Convolutional architectures have emerged as powerful alternatives to Transformers for sequence modeling. The primary advantage is that they offer improved theoretical sequence length complexity by leveraging the Fast Fourier Transform (FFT). However, this theoretical improvement does not always meaningfully land in practice. One critical obstacle is that applying standard FFTs is not amenable to the large-scale training pipeline wherein data is packed from different sources into a single sequence for hardware efficiency. Indeed, standard FFT algorithms are not easily amenable to document packing. Existing workarounds suffer from severe inefficiencies, crippling the practical performance of convolutional architectures. We close this gap with RubiConv, a novel algorithm for performing hardware-efficient, boundary-respecting convolutions on packed sequences. Extensive experiments show that RubiConv achieves significant speedups over both attention and standard FFT-based baselines. This work makes the theoretical efficiency of long convolutional models a practical reality for large-scale, real-world data packing.

Related papers

• MPDiT: Multi-Patch Global-to-Local Transformer Architecture For Efficient Flow Matching and Diffusion Model [6.401952350014671]
  Diffusion Transformers (DiTs) have become widely used in diffusion and flow-matching models.<n>DiTs process the same number of patchified tokens in every block, leading to relatively heavy computation during training process.<n>We introduce a multi-patch transformer design in which early blocks operate on larger patches to capture coarse global context, while later blocks use smaller patches to refine local details.
  arXiv  Detail & Related papers  (2026-03-27T12:30:10Z)
• TNT: Improving Chunkwise Training for Test-Time Memorization [62.78875147721906]
  Recurrent neural networks (RNNs) with deep test-time memorization modules, such as Titans and TTT, represent a promising, linearly-scaling paradigm distinct from Transformers.<n>We introduce TNT, a novel training paradigm that decouples training efficiency from inference performance through a two-stage process.<n>TNT achieves a substantial acceleration in training speed-up to 17 times faster than the most accurate baseline configuration.
  arXiv  Detail & Related papers  (2025-11-10T17:45:09Z)
• Memory-Efficient Training with In-Place FFT Implementation [5.474695910716561]
  Existing implementations, including standard FFT and real FFT, cannot achieve true in-place computation.<n>We propose the first real-domain, fully in-place FFT framework (rdFFT) that preserves input-output memory space consistency.
  arXiv  Detail & Related papers  (2025-11-03T09:36:11Z)
• ParaFormer: Shallow Parallel Transformers with Progressive Approximation [14.82319078008725]
  ParaFormer is a shallow Transformer architecture designed for true parallelism in both structure and computation.<n>Our theoretical analysis shows that their performance relies on inter-layer collaboration for progressive approximation, rather than depth itself.<n>ParaFormer supports up to 15.07x model compression and facilitates model expansion for adaptive continuous learning.
  arXiv  Detail & Related papers  (2025-10-17T08:28:26Z)
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification [61.607788999847564]
  We present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM)<n>We reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model.<n>We propose Dynamic Fine-Tuning (DFT), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token.
  arXiv  Detail & Related papers  (2025-08-07T17:59:04Z)
• Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning [50.05207363001145]
  Efficient Fine-Tuning (PEFT) methods achieve performance comparable to Full Fine-Tuning (FFT)<n>We compare the characteristics of PEFT and FFT in terms of representational capacity and robustness based on optimization theory.<n>Experiments on 15 datasets encompassing classification, generation, reasoning, instruction fine-tuning tasks and 11 adversarial test sets validate our theories.
  arXiv  Detail & Related papers  (2025-05-28T13:35:12Z)
• UFT: Unifying Supervised and Reinforcement Fine-Tuning [27.786964046329455]
  We propose Unified Fine-Tuning (UFT), a novel post-training paradigm that unifies SFT and RFT into a single, integrated process.<n>UFT enables the model to effectively explore solutions while incorporating informative supervision signals.<n>We theoretically prove that UFT breaks RFT's inherent exponential sample complexity bottleneck.
  arXiv  Detail & Related papers  (2025-05-22T17:53:57Z)
• Sliding Window Attention Training for Efficient Large Language Models [55.56483740523027]
  We introduce SWAT, which enables efficient long-context handling via Sliding Window Attention Training.<n>This paper first attributes the inefficiency of Transformers to the attention sink phenomenon.<n>We replace softmax with the sigmoid function and utilize a balanced ALiBi and Rotary Position Embedding for efficient information compression and retention.
  arXiv  Detail & Related papers  (2025-02-26T05:31:44Z)
• Fast Trainable Projection for Robust Fine-Tuning [36.51660287722338]
  Robust fine-tuning aims to achieve competitive in-distribution (ID) performance. Projection-based fine-tuning has been successfully used in robust fine-tuning. Fast Trainable Projection is a new projection-based fine-tuning algorithm.
  arXiv  Detail & Related papers  (2023-10-29T22:52:43Z)
• Softmax-free Linear Transformers [90.83157268265654]
  Vision transformers (ViTs) have pushed the state-of-the-art for visual perception tasks. Existing methods are either theoretically flawed or empirically ineffective for visual recognition. We propose a family of Softmax-Free Transformers (SOFT)
  arXiv  Detail & Related papers  (2022-07-05T03:08:27Z)
• DFTpy: An efficient and object-oriented platform for orbital-free DFT simulations [55.41644538483948]
  In this work, we present DFTpy, an open source software implementing OFDFT written entirely in Python 3. We showcase the electronic structure of a million-atom system of aluminum metal which was computed on a single CPU. DFTpy is released under the MIT license.
  arXiv  Detail & Related papers  (2020-02-07T19:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.