ReLoop-UME: Recurrent Depth with Learnable Retrieval Registers for Universal Multimodal Embedding

• URL: http://arxiv.org/abs/2607.28751v1
• Date: Thu, 30 Jul 2026 18:18:03 GMT
• Title: ReLoop-UME: Recurrent Depth with Learnable Retrieval Registers for Universal Multimodal Embedding
• Abstract summary: Universal multimodal embedding maps heterogeneous multimodal inputs into a shared embedding space.<n>We propose ReLoop-UME, which executes the early layers once, recurrently reuses a parameter-shared retrieval-forming block, and applies the final mapping layers after the last loop.<n>On MMEB-V2 and MRMR, ReLoop-UME consistently improves retrieval across different backbones while running 44.9x faster than UME-R1 and 1.5x faster than PLUME.
• Score: 13.13700187820926
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Universal multimodal embedding (UME) maps heterogeneous multimodal inputs into a shared embedding space. Existing UME models either form embeddings through single forward encoding or add computation through explicit rationale tokens and latent autoregressive states. Although token expansion can improve complex matching, serial generation increases retrieval latency and makes the final embedding depend on generated intermediate states. This raises a different question: can useful computation be expanded along model depth while keeping the token workspace fixed? We analyze positive-negative similarity separation at every layer of independently trained UME models and observe a shared progression: early layers contextualize multimodal inputs, a contiguous middle-to-late stage forms retrieval-discriminative features, and the final layers map them into the embedding space. Based on this finding, we propose ReLoop-UME, which executes the early layers once, recurrently reuses a parameter-shared retrieval-forming block, and applies the final mapping layers after the last loop. Learnable Retrieval Registers provide persistent retrieval-specific states that accumulate and exchange evidence across loops, with the final register serving as the embedding readout. On MMEB-V2 and MRMR, ReLoop-UME consistently improves retrieval across different backbones while running 44.9x faster than UME-R1 and 1.5x faster than PLUME.

Related papers

• UEmbed: Unified Sparse and Dense Multimodal Embeddings [56.1131203949042]
  We introduce UEmbed, a decoder-only multimodal embedding model.<n>It produces both sparse lexical and dense representations in one causal forward pass.<n>We train UEmbed at 2B, 4B, and 9B scales, and it outperforms multimodal embedding models trained on publicly available data.<n>Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.
  arXiv  Detail & Related papers  (2026-08-03T17:54:11Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning [14.508049757681862]
  LLM-powered coding agents spend the majority of their token budget reading repository files.<n>We show that this formulation creates a modeling bottleneck.<n>We propose LaMR, a structured pruning framework that decomposes code relevance into two interpretable quality dimensions.
  arXiv  Detail & Related papers  (2026-05-14T18:30:10Z)
• PLUME: Latent Reasoning Based Universal Multimodal Embedding [52.35354073629127]
  Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model.<n>Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings.<n>We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states.
  arXiv  Detail & Related papers  (2026-04-02T14:04:53Z)
• Attention Residuals [38.59138244826294]
  Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights.<n>We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs.
  arXiv  Detail & Related papers  (2026-03-16T09:32:21Z)
• ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding [37.86179431483446]
  Autoregressive models (ARMs) are hindered by slow sequential inference.<n>We introduce ReFusion, a novel masked diffusion model that achieves superior performance and efficiency.<n>ReFusion bridges the performance gap to strong ARMs while maintaining a 2.33$times$ average speedup.
  arXiv  Detail & Related papers  (2025-12-15T17:41:19Z)
• Recurrence Meets Transformers for Universal Multimodal Retrieval [59.92546492752452]
  ReT-2 is a unified retrieval model that supports multimodal queries composed of both images and text.<n>We evaluate ReT-2 on the challenging M2KR and M-BEIR benchmarks across different retrieval configurations.<n>When integrated into retrieval-augmented generation pipelines, ReT-2 also improves downstream performance on Encyclopedic-VQA and InfoSeek datasets.
  arXiv  Detail & Related papers  (2025-09-10T18:00:29Z)
• Continual Referring Expression Comprehension via Dual Modular Memorization [133.46886428655426]
  Referring Expression (REC) aims to localize an image region of a given object described by a natural-language expression. Existing REC algorithms make a strong assumption that training data feeding into a model are given upfront, which degrades its practicality for real-world scenarios. In this paper, we propose Continual Referring Expression (CREC), a new setting for REC, where a model is learning on a stream of incoming tasks. In order to continuously improve the model on sequential tasks without forgetting prior learned knowledge and without repeatedly re-training from a scratch, we propose an effective baseline method named Dual Modular Memorization
  arXiv  Detail & Related papers  (2023-11-25T02:58:51Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.