ORIC: Benchmarking Object Recognition in Incongruous Context for Large Vision-Language Models

• URL: http://arxiv.org/abs/2509.15695v1
• Date: Fri, 19 Sep 2025 07:14:29 GMT
• Title: ORIC: Benchmarking Object Recognition in Incongruous Context for Large Vision-Language Models
• Authors: Zhaoyang Li, Zhan Ling, Yuchen Zhou, Hao Su,
• Abstract summary: We introduce the Object Recognition in Incongruous Context Benchmark (ORIC), a novel benchmark that evaluates Large Vision-Language Models (LVLMs)<n>ORIC employs two key strategies: (1) LLM-guided sampling, which identifies objects that are present but contextually incongruous, and (2) CLIP-guided sampling, which detects plausible yet nonexistent objects that are likely to be hallucinated.<n>Our results reveal significant recognition gaps, underscoring the challenges posed by contextual incongruity.
• Score: 28.371365768113648
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (LVLMs) have made significant strides in image caption, visual question answering, and robotics by integrating visual and textual information. However, they remain prone to errors in incongruous contexts, where objects appear unexpectedly or are absent when contextually expected. This leads to two key recognition failures: object misidentification and hallucination. To systematically examine this issue, we introduce the Object Recognition in Incongruous Context Benchmark (ORIC), a novel benchmark that evaluates LVLMs in scenarios where object-context relationships deviate from expectations. ORIC employs two key strategies: (1) LLM-guided sampling, which identifies objects that are present but contextually incongruous, and (2) CLIP-guided sampling, which detects plausible yet nonexistent objects that are likely to be hallucinated, thereby creating an incongruous context. Evaluating 18 LVLMs and two open-vocabulary detection models, our results reveal significant recognition gaps, underscoring the challenges posed by contextual incongruity. This work provides critical insights into LVLMs' limitations and encourages further research on context-aware object recognition.

Related papers

• A Multimodal Depth-Aware Method For Embodied Reference Understanding [56.30142869506262]
  Embodied Reference Understanding requires identifying a target object in a visual scene based on both language instructions and pointing cues.<n>We propose a novel ERU framework that jointly leverages data augmentation, depth-map modality, and a depth-aware decision module.
  arXiv  Detail & Related papers  (2025-10-09T14:32:21Z)
• Memorization $\neq$ Understanding: Do Large Language Models Have the Ability of Scenario Cognition? [10.010113234203208]
  We propose a bi-perspective evaluation framework to assess large language models' scenario cognition.<n>We introduce a novel scenario-based dataset comprising diverse textual descriptions of fictional facts, annotated with scenario elements.<n>Our experiments reveal that current LLMs predominantly rely on superficial memorization, failing to achieve robust semantic scenario cognition.
  arXiv  Detail & Related papers  (2025-09-05T07:30:01Z)
• Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models [10.792834356227118]
  Vision-Language Models (VLMs) excel at identifying and describing objects but struggle with spatial reasoning.<n>Inspired by the dual-pathway (ventral-dorsal) model of human vision, we investigate why VLMs fail spatial tasks despite strong object recognition capabilities.
  arXiv  Detail & Related papers  (2025-03-21T17:51:14Z)
• Towards a Systematic Evaluation of Hallucinations in Large-Vision Language Models [57.58426038241812]
  Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in complex multimodal tasks.<n>These models still suffer from hallucinations when required to implicitly recognize or infer diverse visual entities from images.<n>We propose a novel visual question answering (VQA) benchmark that employs contextual reasoning prompts as hallucination attacks.
  arXiv  Detail & Related papers  (2024-12-29T23:56:01Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models [57.43276586087863]
  Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs. Existing benchmarks are often limited in scope, focusing mainly on object hallucinations. We introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases.
  arXiv  Detail & Related papers  (2024-04-22T04:49:22Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• Reason from Context with Self-supervised Learning [15.16197896174348]
  We propose a new Self-supervised method with external memories for Context Reasoning (SeCo) In both tasks, SeCo outperformed all state-of-the-art (SOTA) SSL methods by a significant margin. Our results demonstrate that SeCo exhibits human-like behaviors.
  arXiv  Detail & Related papers  (2022-11-23T10:02:05Z)
• Exploring Multi-Modal Representations for Ambiguity Detection & Coreference Resolution in the SIMMC 2.0 Challenge [60.616313552585645]
  We present models for effective Ambiguity Detection and Coreference Resolution in Conversational AI. Specifically, we use TOD-BERT and LXMERT based models, compare them to a number of baselines and provide ablation experiments. Our results show that (1) language models are able to exploit correlations in the data to detect ambiguity; and (2) unimodal coreference resolution models can avoid the need for a vision component.
  arXiv  Detail & Related papers  (2022-02-25T12:10:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.