Zero-Shot Referring Expression Comprehension via Vison-Language True/False Verification

• URL: http://arxiv.org/abs/2509.09958v2
• Date: Thu, 06 Nov 2025 02:29:52 GMT
• Title: Zero-Shot Referring Expression Comprehension via Vison-Language True/False Verification
• Abstract summary: We show that a zero-shot workflow, without any REC-specific training, can achieve competitive or superior performance.<n>Our approach reformulates REC as box-wise visual-language verification.<n>We show that workflow design, rather than task-specific pretraining, drives strong zero-shot REC performance.
• Score: 1.5777959896871054
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring Expression Comprehension (REC) is usually addressed with task-trained grounding models. We show that a zero-shot workflow, without any REC-specific training, can achieve competitive or superior performance. Our approach reformulates REC as box-wise visual-language verification: given proposals from a COCO-clean generic detector (YOLO-World), a general-purpose VLM independently answers True/False queries for each region. This simple procedure reduces cross-box interference, supports abstention and multiple matches, and requires no fine-tuning. On RefCOCO, RefCOCO+, and RefCOCOg, our method not only surpasses a zero-shot GroundingDINO baseline but also exceeds reported results for GroundingDINO trained on REC and GroundingDINO+CRG. Controlled studies with identical proposals confirm that verification significantly outperforms selection-based prompting, and results hold with open VLMs. Overall, we show that workflow design, rather than task-specific pretraining, drives strong zero-shot REC performance.

Related papers

• Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker [4.731771934553726]
  Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression.<n>OpenRef is a new benchmark for REC in complex visual and linguistic scenarios.
  arXiv  Detail & Related papers  (2026-05-25T11:05:37Z)
• REC-RL: Referring expression counting via Gaussian and range-based reward optimization [28.33789155610508]
  Referring expression counting (REC) is an intention-driven task that requires context-aware visual reasoning.<n>We propose REC-RL, a reinforcement learning framework that introduces a think-range-answer paradigm to explicitly optimize the visual reasoning process.
  arXiv  Detail & Related papers  (2026-05-15T07:37:45Z)
• Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs [18.414159451507153]
  Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries.<n>Existing Vision-Language Models (VLMs) commonly address zero-shot REC by measuring feature similarities between textual queries and image regions.<n>We propose textbfSGREC, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries.
  arXiv  Detail & Related papers  (2026-03-26T04:05:30Z)
• Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks [65.37131487318273]
  Ref-Adv is a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to identify the target.<n>The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation.<n>Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding.
  arXiv  Detail & Related papers  (2026-02-27T10:47:26Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Refinement Provenance Inference: Detecting LLM-Refined Training Prompts from Model Behavior [58.751981587234916]
  This paper formalizes the Refinement Provenance Inference (RPI) audit task as Refinement Provenance Inference (RPI)<n>We propose RePro, a logit-based framework that fuses teacher-forced likelihood features with logit-ranking signals.<n>During training, RePro learns a transferable representation via shadow fine-tuning, and uses a lightweight linear head to infer provenance on unseen victims without training-data access.
  arXiv  Detail & Related papers  (2026-01-05T10:16:41Z)
• SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models [74.40683913645731]
  Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications.<n>Our work proposes a novel solution treating VLMs as black boxes, leveraging scores without training data or ground truth.<n>Analysis of these prompt scores reveals VLM biases and AND''/OR' signal ambiguities, notably that maximum scores are surprisingly suboptimal compared to second-highest scores.
  arXiv  Detail & Related papers  (2025-02-24T07:15:05Z)
• Self-Calibrated Listwise Reranking with Large Language Models [137.6557607279876]
  Large language models (LLMs) have been employed in reranking tasks through a sequence-to-sequence approach. This reranking paradigm requires a sliding window strategy to iteratively handle larger candidate sets. We propose a novel self-calibrated listwise reranking method, which aims to leverage LLMs to produce global relevance scores for ranking.
  arXiv  Detail & Related papers  (2024-11-07T10:31:31Z)
• Continual Referring Expression Comprehension via Dual Modular Memorization [133.46886428655426]
  Referring Expression (REC) aims to localize an image region of a given object described by a natural-language expression. Existing REC algorithms make a strong assumption that training data feeding into a model are given upfront, which degrades its practicality for real-world scenarios. In this paper, we propose Continual Referring Expression (CREC), a new setting for REC, where a model is learning on a stream of incoming tasks. In order to continuously improve the model on sequential tasks without forgetting prior learned knowledge and without repeatedly re-training from a scratch, we propose an effective baseline method named Dual Modular Memorization
  arXiv  Detail & Related papers  (2023-11-25T02:58:51Z)
• Revisiting Large Language Models as Zero-shot Relation Extractors [8.953462875381888]
  Relation extraction (RE) consistently involves a certain degree of labeled or unlabeled data even if under zero-shot setting. Recent studies have shown that large language models (LLMs) transfer well to new tasks out-of-the-box simply given a natural language prompt. This work focuses on the study of exploring LLMs as zero-shot relation extractors.
  arXiv  Detail & Related papers  (2023-10-08T06:17:39Z)
• A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehension [81.57558029858954]
  We build a one-stage referring expression comprehension network called SimREC. We conduct over 100 experimental trials on three benchmark datasets of REC. With much less training overhead and parameters, SimREC can still achieve better performance than a set of large-scale pre-trained models.
  arXiv  Detail & Related papers  (2022-04-17T03:04:03Z)
• Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation [135.67558811281984]
  We propose a novel Multi-task Collaborative Network (MCN) to achieve a joint learning offerring expression comprehension (REC) and segmentation (RES) In MCN, RES can help REC to achieve better language-vision alignment, while REC can help RES to better locate the referent. We address a key challenge in this multi-task setup, i.e., the prediction conflict, with two innovative designs namely, Consistency Energy Maximization (CEM) and Adaptive Soft Non-Located Suppression (ASNLS)
  arXiv  Detail & Related papers  (2020-03-19T14:25:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.