Isotropic Embedding Perturbations for Robust Vision Language Encoders

• URL: http://arxiv.org/abs/2609.10292v1
• Date: Wed, 09 Sep 2026 15:08:05 GMT
• Title: Isotropic Embedding Perturbations for Robust Vision Language Encoders
• Abstract summary: Aether is a simple plug-in method that applies diffusion-style random perturbations in the embedding space via controlled alpha-mixing.<n>Inspired by feature-space perturbations in language models and image degradation in generative pretraining, Aether induces mild yet effective perturbations.<n>Aether delivers consistent gains over the advanced recipe combining CutMix, Mixup, DropPath, and RandAug.
• Score: 73.92626712781859
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data augmentation is fundamental to training modern deep vision and multimodal models. While individual methods, such as RandAug, CutMix, Mixup, RandErase, and DropPath, offer strong regularization effects, their combined use has saturated in performance due to overlapping functionalities, and aggressive pixel-level manipulations may disrupt delicate cross-modal alignment. This saturation motivates the search for a new augmentation axis within the embedding space rather than the input space. We introduce Aether, a simple plug-in method that applies diffusion-style random perturbations in the embedding space via controlled alpha-mixing, specifically designed to provide isotropic regularization that remains semantically consistent. Inspired by feature-space perturbations in language models and image degradation in generative pretraining, Aether induces mild yet effective perturbations that smooth the representations without compromising the fine-grained structural information required for strong vision-language encoders. Across diverse architectures and across multiple recognition tasks, Aether delivers consistent gains over the advanced recipe combining CutMix, Mixup, DropPath, and RandAug---a level of improvement rarely observed with modern augmentation alternatives. Notably, Aether demonstrates superior effectiveness in multi-modal alignment, succeeding where traditional pixel-space augmentations fail by providing a stable, isotropic regularization signal that respects the integrity of the high-dimensional feature space.

Related papers

• Pixel-Space Diffusion Transformers [29.08838156660683]
  Latent diffusion models (LDMs) enable efficient high-resolution image synthesis by denoising in a VAE-compressed latent space.<n>This paper reviews Pixel-Space Diffusion Transformers (pDiTs) from the perspectives of model architecture, continuous generative mechanisms, and unified multimodal modeling.
  arXiv  Detail & Related papers  (2026-07-20T06:04:08Z)
• Coevolving Representations in Joint Image-Feature Diffusion [13.292849179066993]
  We argue that the representation space guiding diffusion should itself adapt to the generative task.<n>We propose Coevolving Diffusion Representation (CoReDi), a framework in which the semantic representation space evolves during training.<n>Experiments show that CoReDi achieves faster convergence and higher sample quality compared to joint diffusion models operating in fixed representation spaces.
  arXiv  Detail & Related papers  (2026-04-19T15:29:15Z)
• Rethinking Vector Field Learning for Generative Segmentation [50.08025820235397]
  Taming diffusion models for generative segmentation has attracted increasing attention.<n>We revisit diffusion segmentation from the perspective of vector field learning.<n>We propose a principled vector field reshaping strategy that augments the learned vanishing velocity field with a detached distance-aware correction term.<n>This correction introduces both attractive and repulsive interactions, enhancing gradient magnitudes near centroids while preserving the original diffusion training framework.
  arXiv  Detail & Related papers  (2026-03-19T17:58:19Z)
• V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising [65.5867130156805]
  We present V-Co, a systematic study of visual co-denoising in a unified JiT-based framework.<n>Our study reveals four key ingredients for effective visual co-denoising.<n>V-Co outperforms the underlying pixel-space diffusion baseline and strong prior pixel-diffusion methods.
  arXiv  Detail & Related papers  (2026-03-17T17:01:54Z)
• LADR: Locality-Aware Dynamic Rescue for Efficient Text-to-Image Generation with Diffusion Large Language Models [10.535207607235046]
  Locality-Aware Dynamic Rescue (LADR) is a training-free method that expedites inference by exploiting the spatial Markov property of images.<n>LADR prioritizes the recovery of tokens at the ''generation frontier'', regions spatially adjacent to observed pixels.<n>Experiments on four text-to-image generation benchmarks demonstrate that our LADR achieves an approximate 4 x speedup over standard baselines.
  arXiv  Detail & Related papers  (2026-03-13T15:12:41Z)
• Hierarchical Granularity Alignment and State Space Modeling for Robust Multimodal AU Detection in the Wild [12.893641381621459]
  We propose a novel multimodal framework driven by Hierarchical Granularity Alignment and State Space Models.<n>We overcome the receptive field limitations of conventional temporal convolutional networks by introducing a Vision-Mamba architecture.<n>This approach enables temporal modeling with O(N) linear complexity, effectively capturing ultra-long-range dynamics without performance degradation.
  arXiv  Detail & Related papers  (2026-03-11T21:05:59Z)
• IdGlow: Dynamic Identity Modulation for Multi-Subject Generation [23.20674988897558]
  We present IdGlow, a mask-free, progressive two-stage framework built upon Flow Matching diffusion models.<n>In the supervised fine-tuning (SFT) stage, we introduce task-adaptive timestep scheduling aligned with diffusion generative dynamics.<n>In the second stage, we design a Fine-Grained Group-Level Direct Preference Optimization (DPO) with a weighted margin formulation to simultaneously eliminate multi-subject artifacts.
  arXiv  Detail & Related papers  (2026-02-28T11:56:34Z)
• MirrorLA: Reflecting Feature Map for Vision Linear Attention [49.41670925034762]
  Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance.<n>We propose MirrorLA, a geometric framework that substitutes passive truncation with active reorientation.<n>MirrorLA achieves state-of-the-art performance across standard benchmarks, demonstrating that strictly linear efficiency can be achieved without compromising representational fidelity.
  arXiv  Detail & Related papers  (2026-02-04T09:14:09Z)
• Training-Free Generation of Diverse and High-Fidelity Images via Prompt Semantic Space Optimization [50.5332987313297]
  We propose Token-Prompt embedding Space Optimization (TPSO), a training-free and model-agnostic module.<n>TPSO introduces learnable parameters to explore underrepresented regions of the token embedding space, reducing the tendency of the model to repeatedly generate samples from strong modes of the learned distribution.<n>In experiments on MS-COCO and three diffusion backbones, TPSO significantly enhances generative diversity, improving baseline performance from 1.10 to 4.18 points, without sacrificing image quality.
  arXiv  Detail & Related papers  (2025-11-25T00:42:09Z)
• RainDiff: End-to-end Precipitation Nowcasting Via Token-wise Attention Diffusion [64.49056527678606]
  We propose a Token-wise Attention integrated into not only the U-Net diffusion model but also the radar-temporal encoder.<n>Unlike prior approaches, our method integrates attention into the architecture without incurring the high resource cost typical of pixel-space diffusion.<n>Our experiments and evaluations demonstrate that the proposed method significantly outperforms state-of-the-art approaches, robustness local fidelity, generalization, and superior in complex precipitation forecasting scenarios.
  arXiv  Detail & Related papers  (2025-10-16T17:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.