COOCO -- Common Objects Out-of-Context -- Semantic Violation in Scenes: Investigating Multimodal Context in Referential Communication

• URL: http://arxiv.org/abs/2506.22274v1
• Date: Fri, 27 Jun 2025 14:44:45 GMT
• Title: COOCO -- Common Objects Out-of-Context -- Semantic Violation in Scenes: Investigating Multimodal Context in Referential Communication
• Authors: Filippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt,
• Abstract summary: We show that Vision-Language Models (VLMs) learn to rely on scene contexts in a similar way, when generating references to objects.<n>Our findings show that models leverage scene context adaptively, depending on both the semantic relatedness between object and scene and the level of noise.
• Score: 3.829419993067886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Natural scenes provide us with rich contexts for object recognition and reference. In particular, knowing what type of scene one is looking at generates expectations about which objects will occur, and what their spatial configuration should be. Do Vision-Language Models (VLMs) learn to rely on scene contexts in a similar way, when generating references to objects? To address this question, we introduce the \textit{Common Objects Out-of-Context (COOCO)} dataset and test to what extent VLMs rely on scene context to refer to objects under different degrees of scene-object congruency, and different perturbations. Our findings show that models leverage scene context adaptively, depending on both the semantic relatedness between object and scene and the level of noise. In particular, models rely more on context under high target-scene congruence or when objects are degraded. Attention analysis reveals that successful object categorisation involves increased focus on the target in mid-level layers, especially under moderate noise, suggesting that VLMs dynamically balance local and contextual information for reference generation. We make our dataset, code and models available at \href{https://github.com/cs-nlp-uu/scenereg}{https://github.com/cs-nlp-uu/scenereg}.

Related papers

• ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splatting [54.92763171355442]
  ObjectGS is an object-aware framework that unifies 3D scene reconstruction with semantic understanding.<n>We show through experiments that ObjectGS outperforms state-of-the-art methods on open-vocabulary and panoptic segmentation tasks.
  arXiv  Detail & Related papers  (2025-07-21T10:06:23Z)
• ContextHOI: Spatial Context Learning for Human-Object Interaction Detection [24.381821663963898]
  spatial contexts are considered critical in Human-Object Interaction (HOI) recognition.<n>We present a dual-branch framework named ContextHOI, which efficiently captures both object detection features and spatial contexts.<n> ContextHOI achieves state-of-the-art performance on the HICO-DET and v-coco benchmarks.
  arXiv  Detail & Related papers  (2024-12-12T08:21:19Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Resilience through Scene Context in Visual Referring Expression Generation [8.883534683127415]
  We investigate the role of context in Referring Expression Generation (REG) for objects in images. We take a new perspective on scene context in REG and hypothesize that contextual information can be conceived of as a resource that makes REG models more resilient.
  arXiv  Detail & Related papers  (2024-04-18T16:10:38Z)
• Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers [65.51132104404051]
  We introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
  arXiv  Detail & Related papers  (2023-12-13T14:27:45Z)
• CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion [83.30168660888913]
  We present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model.
  arXiv  Detail & Related papers  (2023-05-25T17:39:13Z)
• Semantic Attention Flow Fields for Monocular Dynamic Scene Decomposition [51.67493993845143]
  We reconstruct a neural volume that captures time-varying color, density, scene flow, semantics, and attention information. The semantics and attention let us identify salient foreground objects separately from the background across spacetime. We show that this method can decompose dynamic scenes in an unsupervised way with competitive performance to a supervised method.
  arXiv  Detail & Related papers  (2023-03-02T19:00:05Z)
• Hyperbolic Contrastive Learning for Visual Representations beyond Objects [30.618032825306187]
  We focus on learning representations for objects and scenes that preserve the structure among them. Motivated by the observation that visually similar objects are close in the representation space, we argue that the scenes and objects should instead follow a hierarchical structure.
  arXiv  Detail & Related papers  (2022-12-01T16:58:57Z)
• Deriving Visual Semantics from Spatial Context: An Adaptation of LSA and Word2Vec to generate Object and Scene Embeddings from Images [0.0]
  We develop two approaches for learning object and scene embeddings from annotated images. In the first approach, we generate embeddings from object co-occurrences in whole images, one for objects and one for scenes. In the second approach, rather than analyzing whole images of scenes, we focus on co-occurrences of objects within subregions of an image.
  arXiv  Detail & Related papers  (2020-09-20T08:26:38Z)
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation [50.034189314258356]
  We propose a graph model for video captioning that exploits object interactions in space and time. Our model builds interpretable links and is able to provide explicit visual grounding. To avoid correlations caused by the variable number of objects, we propose an object-aware knowledge distillation mechanism.
  arXiv  Detail & Related papers  (2020-03-31T03:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.