Virtual Width Networks

• URL: http://arxiv.org/abs/2511.11238v2
• Date: Mon, 17 Nov 2025 16:00:09 GMT
• Title: Virtual Width Networks
• Abstract summary: We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without increasing the hidden size.<n>In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction.
• Score: 131.13105775581093
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN decouples representational width from backbone width, expanding the embedding space while keeping backbone compute nearly constant. In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction. The advantage amplifies over training as both the loss gap grows and the convergence-speedup ratio increases, showing that VWN is not only token-efficient but also increasingly effective with scale. Moreover, we identify an approximately log-linear scaling relation between virtual width and loss reduction, offering an initial empirical basis and motivation for exploring virtual-width scaling as a new dimension of large-model efficiency.

Related papers

• Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer [46.50233782149623]
  We study the evolution of hidden-weight spectra in wide neural networks trained by (stochastic) gradient descent.<n>We develop a two-level dynamical mean-field theory (DMFT) that jointly tracks bulk and outlier spectral dynamics for spiked ensembles.
  arXiv  Detail & Related papers  (2026-05-08T15:28:01Z)
• SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration [23.86429472943524]
  We present a training-free acceleration framework that exploits three properties of Visual AutoRegressive attention: strong attention sinks, cross-scale activation similarity, and pronounced locality.<n>Specifically, we dynamically predict the sparse attention pattern of later high-resolution scales from a sparse decision scale, and construct scale self-similar sparse attention via an efficient index-mapping mechanism.<n>Our method achieves a $mathbf1.57times$ speed-up while preserving almost all high-frequency details.
  arXiv  Detail & Related papers  (2026-02-04T09:34:06Z)
• FaST: Efficient and Effective Long-Horizon Forecasting for Large-Scale Spatial-Temporal Graphs via Mixture-of-Experts [49.9321870703948]
  Existing models predominantly focus on short-horizon predictions and suffer from notorious computational costs and memory consumption.<n>We present FaST, an effective and efficient framework based on Mixture-of-Experts (MoEs) for long-horizon and large-scale STG forecasting.<n>FaST is underpinned by two key innovations. First, an adaptive graph agent attention mechanism is proposed to alleviate the computational burden.<n>Second, we propose a new parallel MoE module that replaces traditional feed-forward networks with Gated Linear Units (GLUs)
  arXiv  Detail & Related papers  (2026-01-08T18:00:58Z)
• Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient [52.96232442322824]
  Collaborative Decoding (CoDe) is a novel efficient decoding strategy tailored for the Visual Auto-Regressive ( VAR) framework.<n>CoDe capitalizes on two critical observations: the substantially reduced parameter demands at larger scales and the exclusive generation patterns across different scales.<n>CoDe achieves a 1.7x speedup, slashes memory usage by around 50%, and preserves image quality with only a negligible FID increase from 1.95 to 1.98.
  arXiv  Detail & Related papers  (2024-11-26T15:13:15Z)
• On the Diminishing Returns of Width for Continual Learning [2.9301925522760524]
  We analyze Continual Learning Theory to prove that width is directly related to forgetting in Feed-Forward Networks (FFN) Specifically, we demonstrate that increasing network widths to reduce forgetting yields diminishing returns.
  arXiv  Detail & Related papers  (2024-03-11T03:19:45Z)
• Feature-Learning Networks Are Consistent Across Widths At Realistic Scales [72.27228085606147]
  We study the effect of width on the dynamics of feature-learning neural networks across a variety of architectures and datasets. Early in training, wide neural networks trained on online data have not only identical loss curves but also agree in their point-wise test predictions throughout training. We observe, however, that ensembles of narrower networks perform worse than a single wide network.
  arXiv  Detail & Related papers  (2023-05-28T17:09:32Z)
• Efficient NTK using Dimensionality Reduction [5.025654873456756]
  We show how to obtain guarantees to those obtained by a prior analysis while reducing training and inference resource costs. More generally, our work suggests how to analyze large width networks in which dense linear layers are replaced with a low complexity factorization.
  arXiv  Detail & Related papers  (2022-10-10T16:11:03Z)
• Scalable Visual Transformers with Hierarchical Pooling [61.05787583247392]
  We propose a Hierarchical Visual Transformer (HVT) which progressively pools visual tokens to shrink the sequence length. It brings a great benefit by scaling dimensions of depth/width/resolution/patch size without introducing extra computational complexity. Our HVT outperforms the competitive baselines on ImageNet and CIFAR-100 datasets.
  arXiv  Detail & Related papers  (2021-03-19T03:55:58Z)
• Asymptotics of Wide Convolutional Neural Networks [18.198962344790377]
  We study scaling laws for wide CNNs and networks with skip connections. We find that the difference in performance between finite and infinite width models vanishes at a definite rate with respect to model width.
  arXiv  Detail & Related papers  (2020-08-19T21:22:19Z)
• Joint Multi-Dimension Pruning via Numerical Gradient Update [120.59697866489668]
  We present joint multi-dimension pruning (abbreviated as JointPruning), an effective method of pruning a network on three crucial aspects: spatial, depth and channel simultaneously. We show that our method is optimized collaboratively across the three dimensions in a single end-to-end training and it is more efficient than the previous exhaustive methods.
  arXiv  Detail & Related papers  (2020-05-18T17:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.