Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap

• URL: http://arxiv.org/abs/2607.00606v1
• Date: Wed, 01 Jul 2026 08:31:09 GMT
• Title: Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap
• Abstract summary: ReVisIT is a train-free framework that treats each retrieved image-label pair as a unit of visual thought.<n>On VL-ICL Bench Fast Open MiniImageNet, Qwen3-VL-30B-A3B with ReVisIT reaches 98.5% at 4-shot, statistically indistinguishable from the 72B LLaVA-VisionOne SOTA (98.7%) on this near-saturated task at about 1/2.4 the parameters.
• Score: 6.097306160389619
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work on Thinking with Images makes vision a dynamic part of reasoning, but does so through generation: the model invokes external tools, synthesizes code, or imagines new imagery, each at the cost of a tool protocol, brittle code, or an expensive training pipeline. A fourth route makes vision dynamic without generating anything, by retrieving labeled exemplar images and reasoning over them, yet it remains underexplored despite being train-free. We present ReVisIT, a train-free framework that realizes this retrieval-based route by treating each retrieved image-label pair as a unit of visual thought. ReVisIT combines structured class definitions, per-query multimodal retrieval of exemplars, and alternating user/assistant injection of those exemplars before joint multi-attribute decoding, and degrades gracefully to whichever components a task admits. On VL-ICL Bench Fast Open MiniImageNet, Qwen3-VL-30B-A3B with ReVisIT reaches 98.5% at 4-shot, statistically indistinguishable from the 72B LLaVA-OneVision SOTA (98.7%) on this near-saturated task at about 1/2.4 the parameters, while the same backbone without the scaffold sits at chance. The turns layer alone adds 26.1 points to GPT-4.1 on free-form concept induction (Bongard-OpenWorld), and the full stack yields a 4-6 point macro gain across three backbones on MAAC-Bench, a new license-clean 27-class, 5-attribute benchmark, significant by paired bootstrap on the curator-derived attributes. Component analysis shows that retrieval-plus-turns is the universal lever while structured definitions are need-adaptive, and that 83% of the retrieval gain comes from retrieval quality rather than from the presence of exemplars. MAAC-Bench is released with a rubric-grounded LLM verification protocol that replaces author spot-check on subjective attributes.

Related papers

• Learning to Look Again: Loss-Gap Supervision for Free-form Crop Routing in Vision-Language Models [9.530053392873178]
  Vision-language models (VLMs) fail many detail-centric questions for a concrete reason.<n>Allocating more visual tokens to every query improves some OCR and document cases, but it spends indiscriminately and can disturb tasks that rely on global context.<n>We propose GapSight, a framework for learning visual re-reading.
  arXiv  Detail & Related papers  (2026-08-22T04:10:28Z)
• When Generated Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception [12.968836079025344]
  We propose CERES, a closed-loop multimodal indexing framework that builds a three-level semantic pyramid.<n>It mines implicit concepts via a co-language-aware router, performs scale-routed cross-attention into a lightweight U-Net generator, and verifies coverage by re-indexing the generated image with the same frozen VLM.<n>On four pansharpening benchmarks across seven settings, CERES delivers the new state of the art with the largest gains where scale variation is most extreme.
  arXiv  Detail & Related papers  (2026-08-21T07:28:01Z)
• TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation [74.69068352546073]
  TRACE-Bench is an evaluation tool for multi-reference images spanning diverse artistic styles and real-world subjects.<n>It is built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects.<n>Its formula structure drives an operator-aligned evaluation protocol for per-capability scoring and a diagnostic tree analysis.
  arXiv  Detail & Related papers  (2026-08-17T16:15:50Z)
• Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards [52.42920996842378]
  Most unified large multimodal models (LMMs) that support both visual understanding and image generation rely on curated post-training supervision.<n>We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a solver that answers and evaluates them, and a Generator that synthesizes images.<n>For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning.
  arXiv  Detail & Related papers  (2026-06-25T17:59:57Z)
• COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm [59.26203051651017]
  C-TAO is the first continuously annotated training set for Open-Vocabulary Multi-Object Tracking (OVMOT)<n>For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules.<n>Experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods.
  arXiv  Detail & Related papers  (2026-03-25T07:20:27Z)
• SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read [43.28273039987167]
  We introduce the Visualized-Question (VQ) setting, where text queries are rendered directly onto images.<n>Despite possessing strong OCR capabilities, models suffer a performance degradation of up to 12.7% in the VQ setting.<n>We propose SimpleOCR, a plug-and-play training strategy that imposes a structural constraint on the learning process.
  arXiv  Detail & Related papers  (2026-02-25T21:36:30Z)
• Instance-Level Composed Image Retrieval [34.04479584450632]
  i-CIR is a new evaluation dataset that focuses on an instance-level class definition.<n>Its design and curation process keep the dataset compact to facilitate future research.<n>We leverage pre-trained vision-and-language models (VLMs) in a training-free approach called BASIC.
  arXiv  Detail & Related papers  (2025-10-29T10:57:59Z)
• Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions [81.33113485830711]
  We introduce a vision-free, single-encoder retrieval pipeline for vision-language models.<n>We migrate to a text-to-text paradigm with the assistance of VLLM-generated structured image descriptions.<n>Our approach achieves state-of-the-art zero-shot performance on multiple retrieval and compositionality benchmarks.
  arXiv  Detail & Related papers  (2025-09-23T16:22:27Z)
• One RL to See Them All: Visual Triple Unified Reinforcement Learning [92.90120580989839]
  We propose V-Triune, a Visual Triple Unified Reinforcement Learning system that enables visual reasoning and perception tasks within a single training pipeline.<n>V-Triune comprises triple complementary components: Sample-Level Datashelf (to unify diverse task inputs), Verifier-Level Reward (to deliver custom rewards via specialized verifiers).<n>We introduce a novel Dynamic IoU reward, which provides adaptive, progressive, and definite feedback for perception tasks handled by V-Triune.
  arXiv  Detail & Related papers  (2025-05-23T17:41:14Z)
• Raising the Bar of AI-generated Image Detection with CLIP [50.345365081177555]
  The aim of this work is to explore the potential of pre-trained vision-language models (VLMs) for universal detection of AI-generated images. We develop a lightweight detection strategy based on CLIP features and study its performance in a wide variety of challenging scenarios.
  arXiv  Detail & Related papers  (2023-11-30T21:11:20Z)
• ViewFormer: View Set Attention for Multi-view 3D Shape Understanding [7.39435265842079]
  We present ViewFormer, a model for multi-view 3d shape recognition and retrieval. With only 2 attention blocks and 4.8M learnable parameters, ViewFormer reaches 98.8% recognition accuracy on ModelNet40 for the first time. On the challenging RGBD dataset, our method achieves 98.4% recognition accuracy, which is a 4.1% absolute improvement over the strongest baseline.
  arXiv  Detail & Related papers  (2023-04-29T03:58:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.