e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings

• URL: http://arxiv.org/abs/2601.03666v2
• Date: Fri, 09 Jan 2026 02:24:32 GMT
• Title: e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings
• Abstract summary: We propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf vision-language models into robust omni-modal embedding models.<n>e5-omni combines modality-aware temperature calibration to align similarity scales, a controllable negative curriculum with debiasing to focus on confusing negatives, and batch whitening to better match cross-modal geometry.<n> Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines.
• Score: 91.3041144166326
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern information systems often involve different types of items, e.g., a text query, an image, a video clip, or an audio segment. This motivates omni-modal embedding models that map heterogeneous modalities into a shared space for direct comparison. However, most recent omni-modal embeddings still rely heavily on implicit alignment inherited from pretrained vision-language model (VLM) backbones. In practice, this causes three common issues: (i) similarity logits have modality-dependent sharpness, so scores are not on a consistent scale; (ii) in-batch negatives become less effective over time because mixed-modality batches create an imbalanced hardness distribution; as a result, many negatives quickly become trivial and contribute little gradient; and (iii) embeddings across modalities show mismatched first- and second-order statistics, which makes rankings less stable. To tackle these problems, we propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf VLMs into robust omni-modal embedding models. e5-omni combines three simple components: (1) modality-aware temperature calibration to align similarity scales, (2) a controllable negative curriculum with debiasing to focus on confusing negatives while reducing the impact of false negatives, and (3) batch whitening with covariance regularization to better match cross-modal geometry in the shared embedding space. Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines, and the same recipe also transfers well to other VLM backbones. We release our model checkpoint at https://huggingface.co/Haon-Chen/e5-omni-7B.

Related papers

• OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses [15.379006872674106]
  We introduce D3- Omni, a balanced and decoupled benchmark for diagnosing fine-grained multimodal understanding.<n>Rather than re-generating outputs, which may leak information across dimensions, we fix verified fully positive seeds and derive negatives.<n>The resulting D3 design is Dual-balanced, which helps alleviate negative-sample scarcity and per-dimension label imbalance.
  arXiv  Detail & Related papers  (2026-08-25T07:18:35Z)
• Is the Modality Gap a Bug or a Feature? A Robustness Perspective [12.228713815185705]
  Almost all modern multi-modal models seek an embedding space in which the two modalities are aligned.<n>We show that under certain conditions, minimizing the contrastive loss yields a representation in which the two modalities are separated.<n>We also show that under these conditions the modality gap is monotonically related to robustness.
  arXiv  Detail & Related papers  (2026-03-30T23:41:22Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging [39.696384608482255]
  Singular Value (SVC) is a training-free and data-free post-processing method that quantifies subspace overlap and rescales singular values to restore a balanced spectrum.<n>SVC consistently improves strong merging baselines and achieves state-of-the-art performance across vision and language benchmarks.
  arXiv  Detail & Related papers  (2026-02-05T10:52:36Z)
• UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning [101.62386137855704]
  We present a novel Universal Multimodal Embedding (UniME-V2) model.<n>Our approach first constructs a potential hard negative set through global retrieval.<n>We then introduce the MLLM-as-a-Judge mechanism, which utilizes MLLMs to assess the semantic alignment of query-candidate pairs.<n>These scores serve as a foundation for hard negative mining, mitigating the impact of false negatives and enabling the identification of diverse, high-quality hard negatives.
  arXiv  Detail & Related papers  (2025-10-15T13:07:00Z)
• Dynamic Weighted Combiner for Mixed-Modal Image Retrieval [8.683144453481328]
  Mixed-Modal Image Retrieval (MMIR) as a flexible search paradigm has attracted wide attention. Previous approaches always achieve limited performance, due to two critical factors. We propose a Dynamic Weighted Combiner (DWC) to tackle the above challenges.
  arXiv  Detail & Related papers  (2023-12-11T07:36:45Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• Training Vision-Language Models with Less Bimodal Supervision [37.945348204453154]
  We investigate the extent to which we can reduce the reliance on such parallel data. We analyze the effect of bimodal supervision on three vision-language tasks. We find that on simpler tasks, such as VQAv2 and GQA, one can eliminate bimodal supervision completely, suffering only a minor loss in performance.
  arXiv  Detail & Related papers  (2022-11-01T04:07:11Z)
• VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix [59.25846149124199]
  This paper proposes a data augmentation method, namely cross-modal CutMix. CMC transforms natural sentences from the textual view into a multi-modal view. By attaching cross-modal noise on uni-modal data, it guides models to learn token-level interactions across modalities for better denoising.
  arXiv  Detail & Related papers  (2022-06-17T17:56:47Z)
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning [35.25854322376364]
  We show that different data modalities are embedded at arm's length in their shared representation in multi-modal models such as CLIP. In contrastive learning keeps the different modalities separate by a certain distance, which is influenced by the temperature parameter in the loss function. Our experiments further demonstrate that varying the modality gap distance has a significant impact in improving the model's downstream zero-shot classification performance and fairness.
  arXiv  Detail & Related papers  (2022-03-03T22:53:54Z)
• ANIMC: A Soft Framework for Auto-weighted Noisy and Incomplete Multi-view Clustering [59.77141155608009]
  We propose a novel Auto-weighted Noisy and Incomplete Multi-view Clustering framework (ANIMC) via a soft auto-weighted strategy and a doubly soft regular regression model. ANIMC has three unique advantages: 1) it is a soft algorithm to adjust our framework in different scenarios, thereby improving its generalization ability; 2) it automatically learns a proper weight for each view, thereby reducing the influence of noises; and 3) it aligns the same instances in different views, thereby decreasing the impact of missing instances.
  arXiv  Detail & Related papers  (2020-11-20T10:37:27Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.