Beyond Referring Expressions: Scenario Comprehension Visual Grounding

• URL: http://arxiv.org/abs/2604.02323v1
• Date: Thu, 02 Apr 2026 17:59:08 GMT
• Title: Beyond Referring Expressions: Scenario Comprehension Visual Grounding
• Abstract summary: Existing benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category.<n>We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and context rather than explicit naming.<n>We introduce Referring Scenario (RSC), a benchmark designed for this setting. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories.
• Score: 18.49037321302958
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Existing visual grounding benchmarks primarily evaluate alignment between image regions and literal referring expressions, where models can often succeed by matching a prominent named category. We explore a complementary and more challenging setting of scenario-based visual grounding, where the target must be inferred from roles, intentions, and relational context rather than explicit naming. We introduce Referring Scenario Comprehension (RSC), a benchmark designed for this setting. The queries in this benchmark are paragraph-length texts that describe object roles, user goals, and contextual cues, including deliberate references to distractor objects that often require deep understanding to resolve. Each instance is annotated with interpretable difficulty tags for uniqueness, clutter, size, overlap, and position which expose distinct failure modes and support fine-grained analysis. RSC contains approximately 31k training examples, 4k in-domain test examples, and a 3k out-of-distribution split with unseen object categories. We further propose ScenGround, a curriculum reasoning method serving as a reference point for this setting, combining supervised warm-starting with difficulty-aware reinforcement learning. Experiments show that scenario-based queries expose systematic failures in current models that standard benchmarks do not reveal, and that curriculum training improves performance on challenging slices and transfers to standard benchmarks.

Related papers

• SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models [63.28435103335999]
  We introduce SOCO, a new benchmark for Semantic Object Correspondence.<n>We show that vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories.<n>We also show that LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching.
  arXiv  Detail & Related papers  (2026-05-29T17:58:48Z)
• Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting [17.927293384172003]
  Open-world text-guided class-agnostic counting (CAC) has emerged as a flexible paradigm for counting arbitrary object classes by using natural language prompts.<n>We show that several state-of-the-art CAC models often struggle to determine which object class should be counted based on the given prompt.<n>We propose a new evaluation framework focused on model robustness and trustworthiness.
  arXiv  Detail & Related papers  (2026-05-04T15:55:57Z)
• Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval [56.72697928941446]
  Composed Image Retrieval (CIR) has demonstrated significant potential by enabling flexible multimodal queries that combine a reference image and modification text.<n>We propose Object-Anchored Composed Image Retrieval (OACIR), a novel fine-grained retrieval task that mandates strict instance-level consistency.
  arXiv  Detail & Related papers  (2026-04-07T03:43:01Z)
• Open-Vocabulary vs Supervised Learning Methods for Post-Disaster Visual Scene Understanding [4.918510966192794]
  We present a comparative evaluation of supervised learning and open-vocabulary vision models for post-disaster scene understanding.<n>We focus on semantic segmentation and object detection across multiple datasets, including FloodNet+, RescueNet, DFire, and LADD.<n>The most notable remark across all evaluated benchmarks is that supervised training remains the most reliable approach.
  arXiv  Detail & Related papers  (2026-03-01T23:50:08Z)
• Beyond Open Vocabulary: Multimodal Prompting for Object Detection in Remote Sensing Images [52.7196029918473]
  Open-vocabulary object detection in remote sensing commonly relies on text-only prompting to specify target categories.<n>In practice, this assumption often breaks down in remote sensing scenarios due to task- and application-specific category semantics.<n>We propose RS-MPOD, a multimodal open-vocabulary detection framework that reformulates category specification beyond text-only prompting.
  arXiv  Detail & Related papers  (2026-02-02T11:03:01Z)
• SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation [58.80001825332851]
  Referring Image (RIS) aims to segment the target object in an image given a natural language expression.<n>Recent methods predominantly focus on simple expressions like "red car" or "left girl"
  arXiv  Detail & Related papers  (2025-10-11T10:50:58Z)
• Composed Object Retrieval: Object-level Retrieval via Composed Expressions [71.47650333199628]
  Composed Object Retrieval (COR) is a brand-new task that goes beyond image-level retrieval to achieve object-level precision.<n>We construct COR127K, the first large-scale COR benchmark that contains 127,166 retrieval triplets with various semantic transformations in 408 categories.<n>We also present CORE, a unified end-to-end model that integrates reference region encoding, adaptive visual-textual interaction, and region-level contrastive learning.
  arXiv  Detail & Related papers  (2025-08-06T13:11:40Z)
• The Devil is in Fine-tuning and Long-tailed Problems:A New Benchmark for Scene Text Detection [2.4829769683482437]
  Scene text detection methods often fail to replicate such success in real-world scenarios.<n>We uncover two key factors contributing to this discrepancy through extensive experiments.<n>We propose a Long-Tailed Benchmark (LTB) to evaluate ability to handle a diverse range of long-tailed challenges.
  arXiv  Detail & Related papers  (2025-05-21T15:26:46Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Generalized Few-Shot Semantic Segmentation in Remote Sensing: Challenge and Benchmark [18.636210870172675]
  Few-shot semantic segmentation can encourage deep learning models to learn from few labelled examples for novel classes not seen during the training. The generalized few-shot segmentation setting has an additional challenge which encourages models not only to adapt to the novel classes but also to maintain strong performance on the training base classes. We release the dataset augmenting OpenEarthMap with additional classes labelled for the generalized few-shot evaluation setting.
  arXiv  Detail & Related papers  (2024-09-17T14:20:47Z)
• Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Models [14.899741072838994]
  Recent CLIP-like Vision-Language Models (VLMs), pre-trained on large amounts of image-text pairs, have paved the way to open-vocabulary semantic segmentation.<n>We propose two different approaches to automatically generate, at test time, query-specific textual contrastive concepts.
  arXiv  Detail & Related papers  (2024-07-06T12:18:43Z)
• Location-Aware Self-Supervised Transformers [74.76585889813207]
  We propose to pretrain networks for semantic segmentation by predicting the relative location of image parts. We control the difficulty of the task by masking a subset of the reference patch features visible to those of the query. Our experiments show that this location-aware pretraining leads to representations that transfer competitively to several challenging semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T16:24:29Z)
• UniT: Unified Knowledge Transfer for Any-shot Object Detection and Segmentation [52.487469544343305]
  Methods for object detection and segmentation rely on large scale instance-level annotations for training. We propose an intuitive and unified semi-supervised model that is applicable to a range of supervision.
  arXiv  Detail & Related papers  (2020-06-12T22:45:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.