PLUME: Latent Reasoning Based Universal Multimodal Embedding

• URL: http://arxiv.org/abs/2604.02073v2
• Date: Thu, 09 Apr 2026 04:39:33 GMT
• Title: PLUME: Latent Reasoning Based Universal Multimodal Embedding
• Abstract summary: Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model.<n>Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings.<n>We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states.
• Score: 52.35354073629127
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model. Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings, enabling multimodal large language models to better infer complex query intent. However, explicit CoT incurs substantial inference overhead and can compress rich multimodal evidence into a narrow textual bottleneck. We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states. To support diverse multimodal queries, PLUME further introduces a semantic-anchor-guided transition adapter that steers latent rollout along different reasoning trajectories under the same fixed computation budget. To stabilize training, PLUME adopts a progressive explicit-to-latent curriculum that uses verbalized reasoning only as a temporary training scaffold and gradually transfers this behavior into hidden-state computation, eliminating explicit CoT at inference. On the 78-task MMEB-v2 benchmark, PLUME outperforms strong explicit-CoT UME baselines while reducing reasoning from hundreds of generated tokens to fewer than 10 latent steps, delivering over 30x faster inference. PLUME is especially well suited to retrieval settings where relevant evidence is dense, structurally complex, and difficult to organize through verbalized intermediate rationales, such as video and visual document retrieval. These results show that structured latent computation can preserve the benefits of intermediate reasoning without the overhead of explicit rationale generation, providing a stronger and more efficient paradigm for practical retrieval systems.

Related papers

• Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings [18.705254311226565]
  Retrieval-aware Self-Distillation (RASD) constructs privileged guidance from input-supported evidence that distinguishes the positive item from retrieved hard negatives.<n>Retrieval-adaptive Inference (RAI) uses a retrieval confidence head to estimate the remaining retrieval utility of a partial CoT.<n>ReWAM achieves state-of-the-art retrieval performance while delivering up to 5x the inference throughput of competitive explicit-CoT UME methods.
  arXiv  Detail & Related papers  (2026-09-14T09:48:05Z)
• WhisperRec: Latent Reasoning for Efficient Foundation Recommendation Models [51.88075626243057]
  We propose WhisperRec, an efficient latent reasoning framework for foundation recommendation models (FRMs)<n>WhisperRec compresses teacher-generated Chain-of-Thought (CoT) into learnable latent reasoning tokens.<n>We show that WhisperRec consistently outperforms explicit-CoT methods and conventional baselines.
  arXiv  Detail & Related papers  (2026-07-29T08:48:35Z)
• LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck [28.83441191878705]
  We propose LaME, which formulates embedding-oriented latent reasoning as a weakly supervised information bottleneck.<n>LaME employs K learnable reason tokens as a fixed-capacity bottleneck, completing all reasoning within a single forward pass.<n> Experiments on MMEB-v2 and MRMR show that LaME achieves competitive performance, surpassing some explicit CoT-based models.
  arXiv  Detail & Related papers  (2026-06-11T08:47:25Z)
• Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings [32.702215732070826]
  Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings.<n>Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks.<n>We propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimize generation and embedding through a retrieval-friendly rewrite.
  arXiv  Detail & Related papers  (2026-04-24T06:50:11Z)
• TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval [35.86480813138274]
  Universal Multimodal Retrieval requires unified embedding models capable of interpreting diverse user intents.<n>We introduce TRACE (Task-adaptive Reasoning And Embeddings)<n>TRACE unifies generative reasoning with discriminative representation learning.
  arXiv  Detail & Related papers  (2026-03-03T12:36:39Z)
• CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension [49.6969505536365]
  We propose CREM, with a unified framework that enhances multimodal representations for retrieval while preserving generative ability.<n>CREM achieves state-of-the-art retrieval performance on MMEB while maintaining strong generative performance on multiple comprehension benchmarks.
  arXiv  Detail & Related papers  (2026-02-22T08:09:51Z)
• UniT: Unified Multimodal Chain-of-Thought Test-time Scaling [85.590774707406]
  Unified models can handle both multimodal understanding and generation within a single architecture, yet they typically operate in a single pass without iteratively refining their outputs.<n>We introduce UniT, a framework for multimodal test-time scaling that enables a single unified model to reason, verify, and refine across multiple rounds.
  arXiv  Detail & Related papers  (2026-02-12T18:59:49Z)
• Multi-Path Collaborative Reasoning via Reinforcement Learning [54.8518809800168]
  Chain-of-Thought (CoT) reasoning has significantly advanced the problem-solving capabilities of Large Language Models (LLMs)<n>Recent methods attempt to address this by generating soft abstract tokens to enable reasoning in a continuous semantic space.<n>We propose Multi-Path Perception Policy Optimization (M3PO), a novel reinforcement learning framework that explicitly injects collective insights into the reasoning process.
  arXiv  Detail & Related papers  (2025-12-01T10:05:46Z)
• Directional Reasoning Injection for Fine-Tuning MLLMs [51.53222423215055]
  Multimodal large language models (MLLMs) are rapidly advancing, yet their reasoning ability often lags behind that of strong text-only counterparts.<n>Existing methods to bridge this gap rely on supervised fine-tuning over large-scale multimodal reasoning data or reinforcement learning.<n>We propose Directional Reasoning Injection for Fine-Tuning (DRIFT) to solve this problem.
  arXiv  Detail & Related papers  (2025-10-16T18:06:46Z)
• Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction [8.169359626365619]
  We generate a textitchain of thought (CoT) -- a sequence of intermediate reasoning steps. We present a novel conditional prompt distillation method to assimilate the commonsense reasoning ability from large language models. Our approach attains state-of-the-art accuracy and manifests a plethora of advantages concerning interpretability, data efficiency, and cross-domain generalization.
  arXiv  Detail & Related papers  (2023-06-25T04:33:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.