Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers

• URL: http://arxiv.org/abs/2607.00774v1
• Date: Wed, 01 Jul 2026 11:00:09 GMT
• Title: Soft Mixture-of-Recursions: Going Deeper with Recursive Vision Transformers
• Abstract summary: We propose Soft Mixture-of-Recursions (SoftMoR) and its Vision Transformer instantiation, Soft Recursive Vision Transformer (SR-ViT)<n>On ImageNet-1K, increasing recursion depth from 1 to 4 improves SR-ViT-S top-1 accuracy from 79.83% to 82.48% with only 1.7M additional parameters.<n>These results demonstrate that SoftMoR provides a parameter-efficient path to deeper and stronger Vision Transformers through recursion.
• Score: 4.38301148531795
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent recursive Transformer studies have primarily reused shared parameters across computation steps to construct compact, parameter-efficient models. In this work, we leverage recursion to build effectively deeper Transformers with stronger representational capacity. However, in Vision Transformers, simply increasing recursion depth does not reliably improve performance, as existing recursive approaches do not fully utilize the intermediate representations produced throughout recursive computation. We propose Soft Mixture-of-Recursions (SoftMoR) and its Vision Transformer instantiation, Soft Recursive Vision Transformer (SR-ViT). SoftMoR learns token-wise mixture weights to softly combine outputs from all recursion steps, allowing intermediate representations to be utilized in a learnable and flexible way. Across diverse vision tasks, SR-ViT consistently improves as recursion depth increases with minimal parameter overhead. On ImageNet-1K, increasing recursion depth from 1 to 4 improves SR-ViT-S top-1 accuracy from 79.83% to 82.48% with only 1.7M additional parameters, outperforming the substantially larger DeiT-B while using approximately 27% of its parameters. These results demonstrate that SoftMoR provides a parameter-efficient path to deeper and stronger Vision Transformers through recursion.

Related papers

• Recursive Scaling in Masked Diffusion Models [67.93066069475753]
  Masked diffusion models (MDMs) have recently emerged as a promising paradigm for sequence generation.<n>We introduce Recursive Masked Diffusion Models (R-MDMs) to increase model depth without increasing parameter count.<n>Across structured generation tasks, including Sudoku and Countdown, we show that R-MDMs achieve substantially improved parameter efficiency.
  arXiv  Detail & Related papers  (2026-06-16T15:02:49Z)
• Déjà View: Looping Transformers for Multi-View 3D Reconstruction [109.98721386523641]
  Recent feed-forward 3D reconstruction transformers have scaled to over a billion parameters.<n>We show that model depth partially buys iteration, paid for inefficiently in unique parameters, and instead make that iteration explicit in architecture.<n>Our model, DéjView, applies a single looped transformer block recurrently to per-view features for K refinement steps.
  arXiv  Detail & Related papers  (2026-05-28T16:47:48Z)
• Recursive Multi-Agent Systems [106.82473907888497]
  We introduce RecursiveMAS, a multi-agent framework that casts the entire system as a unified latent-space computation.<n>RecursiveMAS connects heterogeneous agents as a collaboration loop through the lightweight RecursiveLink module.<n> Theoretical analyses of runtime complexity and learning dynamics establish that RecursiveMAS is more efficient than standard text-based MAS.
  arXiv  Detail & Related papers  (2026-04-28T17:59:34Z)
• Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models [63.47909317137073]
  Large Multimodal Models (LMMs) have achieved remarkable success in vision-language computation tasks.<n>But their vast parameter counts are often underutilized during both training and inference.<n>We propose RecursiveVLM, a recursive Transformer architecture tailored for LMMs.
  arXiv  Detail & Related papers  (2026-02-09T17:58:23Z)
• MeSH: Memory-as-State-Highways for Recursive Transformers [23.995570647573484]
  Recursive models with fewer parameters often lag behind non-recursive counterparts under matched compute.<n>By probing hidden states, we trace this performance gap to two primary bottlenecks.<n>We introduce a Memory-as-State-Highways scheme, which externalizes state management into an explicit memory buffer.
  arXiv  Detail & Related papers  (2025-10-09T03:23:38Z)
• MOR-VIT: Efficient Vision Transformer with Mixture-of-Recursions [1.0411839100853515]
  MoR-ViT is a novel vision transformer framework that incorporates a token-level dynamic recursion mechanism.<n>Experiments on ImageNet-1K and transfer benchmarks demonstrate that MoR-ViT achieves state-of-the-art accuracy with up to 70% parameter reduction and 2.5x inference acceleration.
  arXiv  Detail & Related papers  (2025-07-29T12:46:36Z)
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation [61.67090981767583]
  We introduce Mixture-of-Recursions (MoR), a unified framework that combines the two axes of efficiency inside a single Recursive Transformer.<n>MoR reuses a shared stack of layers across recursion steps to achieve parameter efficiency, while lightweight routers enable adaptive token-level thinking.<n>We also propose a KV sharing variant that reuses KV pairs from the first recursion, specifically designed to further decrease memory footprint.
  arXiv  Detail & Related papers  (2025-07-14T17:49:00Z)
• Diversity-Guided MLP Reduction for Efficient Large Vision Transformers [62.33249256133204]
  Transformer models achieve excellent scaling property, where the performance is improved with the increment of model capacity.<n>Large-scale model parameters lead to an unaffordable cost of computing and memory.<n>We propose a Diversity-Guided Reduction (DGMR) method to significantly reduce the parameters of large vision transformers.
  arXiv  Detail & Related papers  (2025-06-10T08:59:27Z)
• Sliced Recursive Transformer [23.899076070924153]
  Recursive operation on vision transformers can improve parameter utilization without involving additional parameters. Our model Sliced Recursive Transformer (SReT) is compatible with a broad range of other designs for efficient vision transformers.
  arXiv  Detail & Related papers  (2021-11-09T17:59:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.