MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

• URL: http://arxiv.org/abs/2606.14747v1
• Date: Fri, 05 Jun 2026 05:42:34 GMT
• Title: MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios
• Abstract summary: We introduce MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios.<n> MMLongEmbed comprises four retrieval tasks spanning multiple context-length ranges, covering text, document, and video modalities.
• Score: 44.67920911003142
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context windows do not necessarily translate into effective comprehension and representation of long-context multimodal inputs, which remains a critical bottleneck for real-world deployment. To address the lack of systematic evaluation in this setting, we introduce MMLongEmbed, the first comprehensive benchmark for evaluating MEMs in long-context scenarios. MMLongEmbed comprises four retrieval tasks spanning multiple context-length ranges, covering text, document, and video modalities. Through extensive evaluation of state-of-the-art models, we find that current architectures rely heavily on superficial feature matching and struggle to capture deep semantic and structural dependencies. We further observe that performance degradation varies systematically with context length and key information placement. Moreover, models exhibit substantially different robustness to redundant contextual information across modalities. For reproducibility, the benchmark and code are publicly available.

Related papers

• MMLDSum-LLM: Multimodal Long-Document Summarization with Visual-Alignment and Keyword-Aware [11.787892687988368]
  We introduce MMLDSum-Bench, a high-quality benchmark for multimodal long-document summarization.<n>We also propose MMLDSum-LLM, a reproducible two-stage training framework.<n>Our approach significantly improves key-information coverage and cross-modal consistency.
  arXiv  Detail & Related papers  (2026-07-30T10:54:09Z)
• HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding [1.8482679687103294]
  HAVEN is a hierarchically aligned multimodal benchmark for unified video understanding.<n>Built upon this unified annotation paradigm, we propose a comprehensive evaluation suite spanning summarization, temporal reasoning, multimodal grounding, and saliency ranking.
  arXiv  Detail & Related papers  (2026-05-19T00:48:14Z)
• MMCL-Bench: Multimodal Context Learning from Visual Rules, Procedures, and Evidence [66.12781312894647]
  MMCL-Bench is a benchmark for learning task-local rules, procedures, and empirical patterns from visual or mixed-modality teaching context.<n>This setting requires models to recover and localize relevant evidence from images, screenshots, manuals, videos, and frame sequences before they can reason over the learned context.<n>We evaluate frontier multimodal models with strict-based scoring and find that current systems remain far from robust multimodal context learning.
  arXiv  Detail & Related papers  (2026-05-12T19:57:37Z)
• Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval [27.493644447594367]
  MCMR (Multi-Conditional Multimodal Retrieval) is a large-scale benchmark designed to evaluate fine-grained, multi-condition cross-modal retrieval under natural-language queries.<n>It spans five product domains: upper and bottom clothing, jewelry, shoes, and furniture.<n>We benchmark a diverse suite of MLLM-based multimodal retrievers and vision-language rerankers to assess their condition-aware reasoning abilities.
  arXiv  Detail & Related papers  (2026-03-01T12:53:47Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional [40.11148315577635]
  We present a large-scale empirical study to quantify dependencies across 23 visual question-answering benchmarks using multi-modal large language models (MLLMs)<n>Our findings show that the reliance on vision, question (text), and their interaction varies significantly, both across and within benchmarks.<n>We discover that numerous benchmarks intended to mitigate text-only biases have inadvertently amplified image-only dependencies.<n>This characterization persists across model sizes, as larger models often use these intra-modality dependencies to achieve high performance that mask an underlying lack of multi-modal reasoning.
  arXiv  Detail & Related papers  (2025-09-27T21:13:29Z)
• Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts [56.7225771305861]
  This paper introduces Multi-Modal Retrieval-Augmented Generation (M$2$RAG), a benchmark designed to evaluate the effectiveness of Multi-modal Large Language Models.<n>The benchmark comprises four tasks: image captioning, multi-modal question answering, multi-modal fact verification, and image reranking.<n>To enhance the context utilization capabilities of MLLMs, we also introduce Multi-Modal Retrieval-Augmented Instruction Tuning (MM-RAIT)
  arXiv  Detail & Related papers  (2025-02-24T16:25:25Z)
• VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation [100.06122876025063]
  This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings.<n>We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG.
  arXiv  Detail & Related papers  (2024-12-14T06:24:55Z)
• Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning [68.43706033424378]
  This study introduces an innovative method designed to increase in-context text length in large language models (MLLMs) efficiently. We present Visualized In-Context Text Processing (VisInContext), which processes long in-context text using visual tokens. This technique significantly reduces GPU memory usage and floating point operations (FLOPs) for both training and inferenceing stage.
  arXiv  Detail & Related papers  (2024-06-04T17:59:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.