SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models

• URL: http://arxiv.org/abs/2509.15432v1
• Date: Thu, 18 Sep 2025 21:11:13 GMT
• Title: SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models
• Authors: Thong Nguyen, Yibin Lei, Jia-Huei Ju, Andrew Yates,
• Abstract summary: Visual Document Retrieval (VDR) typically operates as text-to-image retrieval using specialized bi-encoders trained to directly embed document images.<n>We revisit a zero-shot generate-and-encode pipeline: a vision-language model first produces a detailed textual description of each document image.<n>On the ViDoRe-v2 benchmark, the method reaches 63.4% nDCG@5, surpassing the strongest specialised multi-vector visual document encoder.
• Score: 17.85605201420847
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual Document Retrieval (VDR) typically operates as text-to-image retrieval using specialized bi-encoders trained to directly embed document images. We revisit a zero-shot generate-and-encode pipeline: a vision-language model first produces a detailed textual description of each document image, which is then embedded by a standard text encoder. On the ViDoRe-v2 benchmark, the method reaches 63.4% nDCG@5, surpassing the strongest specialised multi-vector visual document encoder. It also scales better to large collections and offers broader multilingual coverage. Analysis shows that modern vision-language models capture complex textual and visual cues with sufficient granularity to act as a reusable semantic proxy. By offloading modality alignment to pretrained vision-language models, our approach removes the need for computationally intensive text-image contrastive training and establishes a strong zero-shot baseline for future VDR systems.

Related papers

• ModernVBERT: Towards Smaller Visual Document Retrievers [8.752477008109844]
  ModernVBERT is a compact vision-language encoder that outperforms models up to 10 times larger when finetuned on document retrieval tasks.<n>We measure the impact of attention masking, image resolution, modality alignment data regimes, and late interaction centered contrastive objectives which emerge as central performance factors.
  arXiv  Detail & Related papers  (2025-10-01T17:41:17Z)
• Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions [81.33113485830711]
  We introduce a vision-free, single-encoder retrieval pipeline for vision-language models.<n>We migrate to a text-to-text paradigm with the assistance of VLLM-generated structured image descriptions.<n>Our approach achieves state-of-the-art zero-shot performance on multiple retrieval and compositionality benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T16:22:27Z)
• Visual Lexicon: Rich Image Features in Language Space [99.94214846451347]
  ViLex simultaneously captures rich semantic content and fine visual details.<n>ViLex generates tokens optimized for reconstructing input images using a frozen text-to-image (T2I) diffusion model.<n>As an image embedding in the language space, ViLex tokens leverage the compositionality of natural languages.
  arXiv  Detail & Related papers  (2024-12-09T18:57:24Z)
• UNIT: Unifying Image and Text Recognition in One Vision Encoder [51.140564856352825]
  UNIT is a novel training framework aimed at UNifying Image and Text recognition within a single model. We show that UNIT significantly outperforms existing methods on document-related tasks. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment.
  arXiv  Detail & Related papers  (2024-09-06T08:02:43Z)
• UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model [108.85584502396182]
  We propose UReader, a first exploration of universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM) By leveraging the shallow text recognition ability of the MLLM, we only finetuned 1.2% parameters. Our single model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks.
  arXiv  Detail & Related papers  (2023-10-08T11:33:09Z)
• DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents [18.080447065002392]
  We propose DocumentCLIP to enforce vision-language pretraining models to comprehend the interaction between images and longer text within documents. Our model is beneficial for the real-world multimodal document understanding like news article, magazines, product descriptions, which contain linguistically and visually richer content.
  arXiv  Detail & Related papers  (2023-06-09T23:51:11Z)
• Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment [81.73717488887938]
  Language-Quantized AutoEncoder (LQAE) learns to align text-image data in an unsupervised manner by leveraging pretrained language models. LQAE learns to represent similar images with similar clusters of text tokens, thereby aligning these two modalities without the use of aligned text-image pairs. This enables few-shot image classification with large language models (e.g., GPT-3) as well as linear classification of images based on BERT text features.
  arXiv  Detail & Related papers  (2023-02-02T06:38:44Z)
• Vision-Language Pre-Training for Boosting Scene Text Detectors [57.08046351495244]
  We specifically adapt vision-language joint learning for scene text detection. We propose to learn contextualized, joint representations through vision-language pre-training. The pre-trained model is able to produce more informative representations with richer semantics.
  arXiv  Detail & Related papers  (2022-04-29T03:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.