PUMA: Post-Hoc Sparsification of Universal Multimodal Embeddings for Efficient Retrieval

• URL: http://arxiv.org/abs/2608.25780v1
• Date: Wed, 26 Aug 2026 13:26:39 GMT
• Title: PUMA: Post-Hoc Sparsification of Universal Multimodal Embeddings for Efficient Retrieval
• Abstract summary: We introduce PUMA, a sparse autoencoder recipe that maps universal multimodal embeddings to compact sparse codes without retraining the backbone.<n>We evaluate on five benchmarks covering text-to-image and composed image retrieval.<n>Puma reduces vector storage by 8-16x (FP32) and is up to 25x faster than exact dense scoring on larger candidate pools.
• Score: 50.986807025432476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Universal multimodal embedders enable retrieval across text, image, and combined queries, but their dense representations incur high memory and inference costs. Post-hoc sparsification could reduce these costs but remains underexplored for multimodal retrieval. We introduce PUMA, a sparse autoencoder recipe that maps universal multimodal embeddings to compact sparse codes without retraining the backbone: a pretraining stage preserves dense dot-product geometry, after which the sparse encoder is fine-tuned for retrieval. We evaluate on five benchmarks covering text-to-image and composed image retrieval. On Qwen3-VL-Embedding-2B, PUMA is statistically indistinguishable from or improves over dense retrieval on four of five datasets. We further identify two failure modes of post-hoc sparsification: insufficient pre-TopK support and retrieval-misaligned active support. PUMA reduces vector storage by 8-16x (FP32) and is up to 25x faster than exact dense scoring on larger candidate pools, enabling efficient multimodal retrieval.

Related papers

• UEmbed: Unified Sparse and Dense Multimodal Embeddings [56.1131203949042]
  We introduce UEmbed, a decoder-only multimodal embedding model.<n>It produces both sparse lexical and dense representations in one causal forward pass.<n>We train UEmbed at 2B, 4B, and 9B scales, and it outperforms multimodal embedding models trained on publicly available data.<n>Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.
  arXiv  Detail & Related papers  (2026-08-03T17:54:11Z)
• ReLoop-UME: Recurrent Depth with Learnable Retrieval Registers for Universal Multimodal Embedding [13.13700187820926]
  Universal multimodal embedding maps heterogeneous multimodal inputs into a shared embedding space.<n>We propose ReLoop-UME, which executes the early layers once, recurrently reuses a parameter-shared retrieval-forming block, and applies the final mapping layers after the last loop.<n>On MMEB-V2 and MRMR, ReLoop-UME consistently improves retrieval across different backbones while running 44.9x faster than UME-R1 and 1.5x faster than PLUME.
  arXiv  Detail & Related papers  (2026-07-30T18:18:03Z)
• No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval [51.43543998583709]
  Single-stage Sparse Retrieval (SSR) is a paradigm shift that replaces expensive clustering with efficient sparse coding.<n>It reduces indexing time by 15x compared to ColBERTv2, halves retrieval latency, and simultaneously improves retrieval performance.
  arXiv  Detail & Related papers  (2026-05-28T15:53:34Z)
• Very Efficient Listwise Multimodal Reranking for Long Documents [7.558933130117637]
  We propose ZipRerank, a highly efficient listwise multimodal reranker.<n>It reduces input length via a lightweight query-image early interaction mechanism.<n>Experiments on the MMDocIR benchmark show that ZipRerank matches or surpasses state-of-the-art multimodal rerankers.
  arXiv  Detail & Related papers  (2026-05-12T09:45:59Z)
• MINER: Mining Multimodal Internal Representation for Efficient Retrieval [30.225207646393542]
  MINER (Mining Multimodal Internal RepreseNtation for Efficient Retrieval) is a lightweight plug-in module that probes and fuses internal signals across transformer layers into a single compact embedding.<n>Across ViDoRe V1/V2/V3, MINER outperforms existing dense single-vector retrievers on the majority of benchmarks, with up to 4.5% nDCG@5 improvement over its corresponding backbone.
  arXiv  Detail & Related papers  (2026-05-07T15:51:06Z)
• Multivector Reranking in the Era of Strong First-Stage Retrievers [11.098422338598454]
  We reproduce several state-of-the-art multivector retrieval methods on two publicly available datasets.<n>We show that replacing the token-level gather phase with a single-vector document retriever produces a smaller and more semantically coherent candidate set.<n>Our two-stage approach achieves over $24times$ speedup over the state-of-the-art multivector retrieval systems, while maintaining comparable or superior retrieval quality.
  arXiv  Detail & Related papers  (2026-01-08T18:22:18Z)
• VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing [59.73939718087177]
  Single-encoder vision-language model trained contrastively to embed interleaved inputs in a unified vector space.<n>VLM2GeoVec unifies scalable retrieval with region-level spatial reasoning, enabling cohesive multimodal analysis in remote sensing.
  arXiv  Detail & Related papers  (2025-12-12T11:39:35Z)
• Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control [66.78146440275093]
  Learned retrieval (LSR) is a family of neural methods that encode queries and documents into sparse lexical vectors. We explore the application of LSR to the multi-modal domain, with a focus on text-image retrieval. Current approaches like LexLIP and STAIR require complex multi-step training on massive datasets. Our proposed approach efficiently transforms dense vectors from a frozen dense model into sparse lexical vectors.
  arXiv  Detail & Related papers  (2024-02-27T14:21:56Z)
• Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering [87.32442219333046]
  We propose a simple and resource-efficient method to pretrain the paragraph encoder. Our method outperforms an existing dense retrieval method that uses 7 times more computational resources for pretraining.
  arXiv  Detail & Related papers  (2020-04-30T18:09:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.