VISReg: Variance-Invariance-Sketching Regularization for JEPA training

• URL: http://arxiv.org/abs/2606.02572v1
• Date: Mon, 01 Jun 2026 17:58:01 GMT
• Title: VISReg: Variance-Invariance-Sketching Regularization for JEPA training
• Abstract summary: Self-supervised learning methods prevent embedding collapse via modelings or explicit regularization of the embedding space.<n>We propose Variance-Invariance-Sketching Regularization (VISReg)<n>VISReg replaces covariance with a Sliced-Wasserstein-based sketching objective that enforces full distributional shape.<n>We show that VISReg scales linearly, outperforms existing regularization on low-quality datasets, and is resilient to long-tailed and low-rank regimes.
• Score: 21.894781659564817
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning methods prevent embedding collapse via modeling heuristics or explicit regularization of the embedding space. Among the latter, VICReg decomposes regularization into variance and covariance objectives, offering flexibility and interpretability. However, covariance captures only second-order statistics -- encouraging decorrelation but failing to enforce the full distributional shape needed for stable training. Sketching-based methods such as SIGReg address this by aligning embeddings to an isotropic Gaussian, but lack flexibility and suffer from vanishing gradients under collapse. We propose Variance-Invariance-Sketching Regularization (VISReg), which replaces covariance with a Sliced-Wasserstein-based sketching objective that enforces full distributional shape, while retaining a variance term for scale control. By decoupling scale and shape, VISReg combines VICReg's flexibility with the distributional rigor of sketching methods, providing robust gradients even under collapse. We show that VISReg scales linearly, outperforms existing regularization on low-quality datasets, and is resilient to long-tailed and low-rank regimes. Pre-trained on ImageNet-1K, VISReg achieves state-of-the-art performance on out-of-distribution datasets. Pre-trained on ImageNet-22K, it matches DINOv2's OOD performance despite the latter using 10x more data (LVD-142M). Project and code: https://haiyuwu.github.io/visreg.

Related papers

• Temporally Centered SIGReg Improves Multi-Task LeWorldModel Learning: From Analysis to Method [35.288206438331436]
  We show that marginal Gaussianization compresses the separation between task-dependent latent clusters.<n>To address this problem, we apply SIGReg to temporally centered residuals rather than to the latent marginal distribution.<n>Our method improves downstream success on the long-horizon suite by 1.7x and raises the average success rate across four suites from 53.2% to 73.6%.
  arXiv  Detail & Related papers  (2026-07-29T13:54:52Z)
• Weak-SIGReg: Covariance Regularization for Stable Deep Learning [0.0]
  This work adopts Sketched Isotropic Gaussian Regularization ( SIGReg), recently introduced in the LeJEPA self-supervised framework, and repurposes it as a general optimization stabilizer for supervised learning.<n>Inspired by interacting particle systems, representation collapse is viewed as drift; SIGReg constrains the representation density towards an isotropic Gaussian, mitigating this drift.<n> Empirically, SIGReg recovers the training of a ViT on CIFAR-100 from a collapsed 20.73% to 72.02% accuracy without architectural hacks and significantly improves the convergence of deep vanillas trained with pure SGD. Code is available at
  arXiv  Detail & Related papers  (2026-03-06T05:32:40Z)
• Soft Equivariance Regularization for Invariant Self-Supervised Learning [23.047550451521662]
  Self-supervised learning (SSL) typically learns representations invariant to semantic-preserving augmentations.<n>We propose Soft Equivariance Regularization (SER), a plug-in regularizer that decouples where invariance and equivariance are enforced.<n>SER learns/predicts no per-sample transformation codes/labels, requires no auxiliary transformation-prediction head, and adds only 1.008x training FLOPs.
  arXiv  Detail & Related papers  (2026-03-04T13:36:17Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Self-Supervised Learning by Curvature Alignment [0.15293427903448018]
  CurvSSL is a curvature-regularized self-supervised learning framework.<n>We introduce CurvSSL, a curvature-regularized self-supervised learning framework.<n>We show that explicitly shaping local geometry is a simple and effective complement to purely statistical SSL regularizers.
  arXiv  Detail & Related papers  (2025-11-21T17:22:31Z)
• Compensating Distribution Drifts in Class-incremental Learning of Pre-trained Vision Transformers [27.14203097630326]
  We introduce a latent space transition operator and propose Sequential Learning with Drift Compensation.<n>SLDC aims to align feature distributions across tasks to mitigate the impact of drift.<n>Experiments on standard CIL benchmarks demonstrate that SLDC significantly improves the performance of SeqFT.
  arXiv  Detail & Related papers  (2025-11-13T03:40:54Z)
• Low-Rank Tensor Recovery via Variational Schatten-p Quasi-Norm and Jacobian Regularization [49.85875869048434]
  We propose a CP-based low-rank tensor function parameterized by neural networks for implicit neural representation.<n>To achieve sparser CP decomposition, we introduce a variational Schatten-p quasi-norm to prune redundant rank-1 components.<n>For smoothness, we propose a regularization term based on the spectral norm of the Jacobian and Hutchinson's trace estimator.
  arXiv  Detail & Related papers  (2025-06-27T11:23:10Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Learning GFlowNets from partial episodes for improved convergence and stability [56.99229746004125]
  Generative flow networks (GFlowNets) are algorithms for training a sequential sampler of discrete objects under an unnormalized target density. Existing training objectives for GFlowNets are either local to states or transitions, or propagate a reward signal over an entire sampling trajectory. Inspired by the TD($lambda$) algorithm in reinforcement learning, we introduce subtrajectory balance or SubTB($lambda$), a GFlowNet training objective that can learn from partial action subsequences of varying lengths.
  arXiv  Detail & Related papers  (2022-09-26T15:44:24Z)
• Robust Training of Neural Networks using Scale Invariant Architectures [70.67803417918854]
  In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks. We show that this general approach is robust to rescaling of parameter and loss. We design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam.
  arXiv  Detail & Related papers  (2022-02-02T11:58:56Z)
• VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning [43.96465407127458]
  We introduce VICReg, a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings. VICReg achieves results on par with the state of the art on several downstream tasks.
  arXiv  Detail & Related papers  (2021-05-11T09:53:21Z)
• Simple and Effective Prevention of Mode Collapse in Deep One-Class Classification [93.2334223970488]
  We propose two regularizers to prevent hypersphere collapse in deep SVDD. The first regularizer is based on injecting random noise via the standard cross-entropy loss. The second regularizer penalizes the minibatch variance when it becomes too small.
  arXiv  Detail & Related papers  (2020-01-24T03:44:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.