UEmbed: Unified Sparse and Dense Multimodal Embeddings

• URL: http://arxiv.org/abs/2608.02583v1
• Date: Mon, 03 Aug 2026 17:54:11 GMT
• Title: UEmbed: Unified Sparse and Dense Multimodal Embeddings
• Abstract summary: We introduce UEmbed, a decoder-only multimodal embedding model.<n>It produces both sparse lexical and dense representations in one causal forward pass.<n>We train UEmbed at 2B, 4B, and 9B scales, and it outperforms multimodal embedding models trained on publicly available data.<n>Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.
• Score: 56.1131203949042
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to multimodal settings still relies heavily on auxiliary cross-modal modules. To address these limitations, we introduce UEmbed (Unified Embedding), a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets. Each token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, outperforming multimodal embedding models trained on publicly available data (e.g., RzenEmbed). On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications. Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.

Related papers

• Douyin Multimodal Embedding Model Technical Report [77.9018834072698]
  Multimodal representation learning is a cornerstone of modern AI.<n>We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths.
  arXiv  Detail & Related papers  (2026-08-03T12:31:49Z)
• On the Challenges and Opportunities of Learned Sparse Retrieval for Code [13.929722643500341]
  SPLADE-Code is the first large-scale family of learned sparse retrieval models specialized for code retrieval.<n>We show that SPLADE-Code achieves state-of-the-art performance among retrievers under 1B parameters.
  arXiv  Detail & Related papers  (2026-03-23T14:14:08Z)
• Decomposing multimodal embedding spaces with group-sparse autoencoders [4.817429789586128]
  We propose a new SAE-based approach to multimodal embedding decomposition using cross-modal random masking and group-sparse regularization.<n>We show that, compared to standard SAEs, our approach learns a more multimodal dictionary while reducing the number of dead neurons and improving feature semanticity.
  arXiv  Detail & Related papers  (2026-01-27T20:04:07Z)
• Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline [56.790045049514326]
  Two major forms of deception dominate: human-crafted misinformation and AI-generated content.<n>We propose Unified Multimodal Fake Content Detection (UMFDet), a framework designed to handle both forms of deception.<n>UMFDet achieves robust and consistent performance across both misinformation types, outperforming specialized baselines.
  arXiv  Detail & Related papers  (2025-09-30T09:26:32Z)
• MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction [13.70527493534928]
  We introduce MetaEmbed, a new framework for multimodal retrieval.<n>During training, a fixed number of learnable Meta Tokens are appended to the input sequence.<n>At test-time, their last-layer contextualized representations serve as compact yet expressive multi-vector embeddings.
  arXiv  Detail & Related papers  (2025-09-22T17:59:42Z)
• SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension [77.93156509994994]
  We show how to represent short chunks in a way that is conditioned on a broader context window to enhance retrieval performance.<n>Existing embedding models are not well-equipped to encode such situated context effectively.<n>Our method substantially outperforms state-of-the-art embedding models.
  arXiv  Detail & Related papers  (2025-08-03T23:59:31Z)
• Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control [66.78146440275093]
  Learned retrieval (LSR) is a family of neural methods that encode queries and documents into sparse lexical vectors. We explore the application of LSR to the multi-modal domain, with a focus on text-image retrieval. Current approaches like LexLIP and STAIR require complex multi-step training on massive datasets. Our proposed approach efficiently transforms dense vectors from a frozen dense model into sparse lexical vectors.
  arXiv  Detail & Related papers  (2024-02-27T14:21:56Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• Unsupervised Multimodal Language Representations using Convolutional Autoencoders [5.464072883537924]
  We propose extracting unsupervised Multimodal Language representations that are universal and can be applied to different tasks. We map the word-level aligned multimodal sequences to 2-D matrices and then use Convolutional Autoencoders to learn embeddings by combining multiple datasets. It is also shown that our method is extremely lightweight and can be easily generalized to other tasks and unseen data with small performance drop and almost the same number of parameters.
  arXiv  Detail & Related papers  (2021-10-06T18:28:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.