Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker

• URL: http://arxiv.org/abs/2605.25706v1
• Date: Mon, 25 May 2026 11:05:37 GMT
• Title: Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker
• Abstract summary: Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression.<n>OpenRef is a new benchmark for REC in complex visual and linguistic scenarios.
• Score: 4.731771934553726
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression. Although recent advances in vision-language models have led to substantial improvements in REC tasks, current REC benchmarks often hold simple scenarios and the assumption that each expression maps to a unique object. These limitations hinder the deployment of REC models in open-world environments. To fill this gap, we introduce OpenRef, a new benchmark for REC in complex visual and linguistic scenarios. OpenRef features three key advancements: 1) Diverse visual scenarios: spanning diverse visual domains, including ground views, drone views, dark scenes and adverse weather conditions; 2) Variable target counts: breaking the single-target limitation with multi-target and none-target samples; 3) Rich vocabulary types: incorporating proper nouns, polysemous words and ordinal terms to fit a wider range of expression needs. Furthermore, as traditional metrics are insufficient for open-world setting, we leverage F1 to measure grounding accuracy and propose N3R (Negative Relative Rejection Reliability) to assess relative rejection reliability against negative expressions. Finally, we introduce Multi-task Consistency Checker (MCC), a training-free but plug-and-play strategy that enhances model performance with one click by enforcing consistency self-verification. Extensive experiments demonstrate that this work significantly advances the performance of existing REC models in complex scenarios, paving the way for open-world REC. Project page: https://zongjianwu.github.io/openref

Related papers

• UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery [69.87589293875112]
  Unmanned aerial vehicles (UAVs) increasingly rely on visual grounding capabilities to localize task-relevant targets.<n>Existing referring expression comprehension benchmarks and methods are largely built around text-only queries and single-object outputs.<n>We introduce emphUniversal Referring, a generalized UAV referring task that jointly expands the query modality and the output cardinality.
  arXiv  Detail & Related papers  (2026-07-09T09:09:31Z)
• Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks [65.37131487318273]
  Ref-Adv is a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to identify the target.<n>The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation.<n>Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding.
  arXiv  Detail & Related papers  (2026-02-27T10:47:26Z)
• RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension [45.091078689395864]
  Referring Expression (REC) is a vision-language task that localizes a specific image region based on a textual description.<n>We introduce RefBench-PRO, a comprehensive REC benchmark, which decomposes referring expressions into two core dimensions, i.e., perception and reasoning.<n>We also propose Ref-R1, an RL-based learning scheme, which incorporates Dynamic IoU-based GRPO to improve localization accuracy under increasingly complex reasoning conditions.
  arXiv  Detail & Related papers  (2025-12-06T03:59:21Z)
• SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation [58.80001825332851]
  Referring Image (RIS) aims to segment the target object in an image given a natural language expression.<n>Recent methods predominantly focus on simple expressions like "red car" or "left girl"
  arXiv  Detail & Related papers  (2025-10-11T10:50:58Z)
• KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge [1.5833270109954136]
  We propose KnowDR-REC, built upon real-world knowledge, requiring fine-grained multimodal reasoning across text and image.<n>We evaluate 16 state-of-the-art multimodal models on KnowDR-REC, with experimental results showing that existing MLLMs still struggle with knowledge-driven visual grounding tasks.
  arXiv  Detail & Related papers  (2025-08-12T19:43:44Z)
• Referring Expression Instance Retrieval and A Strong End-to-End Baseline [37.47466772169063]
  Text-Image Retrieval retrieves a target image from a gallery based on an image-level description.<n>Referring Expression localizes a target object within a given image using an instance-level description.<n>We introduce a new task called textbfReferring Expression Instance Retrieval (REIR), which supports both instance-level retrieval and localization.
  arXiv  Detail & Related papers  (2025-06-23T02:28:44Z)
• New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaboration [49.180693704510006]
  Referring Expression (REC) is a cross-modal task that evaluates the interplay of language understanding, image comprehension, and language-to-image grounding.<n>It serves as an essential testing ground for Multimodal Large Language Models (MLLMs)
  arXiv  Detail & Related papers  (2025-02-27T13:58:44Z)
• RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenes [44.34174574185866]
  RefDrone is a REC benchmark for drone scenes.<n> RDAgent is a semi-automated annotation tool for REC tasks.<n>NGDINO is a novel method designed to handle multi-target and no-target cases.
  arXiv  Detail & Related papers  (2025-02-01T10:44:11Z)
• CountGD: Multi-Modal Open-World Counting [54.88804890463491]
  This paper aims to improve the generality and accuracy of open-vocabulary object counting in images.<n>We introduce the first open-world counting model, CountGD, where the prompt can be specified by a text description or visual exemplars or both.
  arXiv  Detail & Related papers  (2024-07-05T16:20:48Z)
• Continual Referring Expression Comprehension via Dual Modular Memorization [133.46886428655426]
  Referring Expression (REC) aims to localize an image region of a given object described by a natural-language expression. Existing REC algorithms make a strong assumption that training data feeding into a model are given upfront, which degrades its practicality for real-world scenarios. In this paper, we propose Continual Referring Expression (CREC), a new setting for REC, where a model is learning on a stream of incoming tasks. In order to continuously improve the model on sequential tasks without forgetting prior learned knowledge and without repeatedly re-training from a scratch, we propose an effective baseline method named Dual Modular Memorization
  arXiv  Detail & Related papers  (2023-11-25T02:58:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.