Contextual inference from single objects in Vision-Language models

• URL: http://arxiv.org/abs/2603.26731v1
• Date: Fri, 20 Mar 2026 13:24:15 GMT
• Title: Contextual inference from single objects in Vision-Language models
• Abstract summary: How much scene context a single object carries is a well-studied question in human scene perception.<n>How this capacity is organized in vision-language models (VLMs) remains poorly understood.<n>We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects.
• Score: 10.367669666212473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How much scene context a single object carries is a well-studied question in human scene perception, yet how this capacity is organized in vision-language models (VLMs) remains poorly understood, with direct implications for the robustness of these models. We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects. Presenting VLMs with single objects on masked backgrounds, we probe their ability to infer both fine-grained scene category and coarse superordinate context (indoor vs. outdoor). We found that single objects support above-chance inference at both levels, with performance modulated by the same object properties that predict human scene categorization. Object identity, scene, and superordinate predictions are partially dissociable: accurate inference at one level neither requires nor guarantees accurate inference at the others, and the degree of coupling differs markedly across models. Mechanistically, object representations that remain stable when background context is removed are more predictive of successful contextual inference. Scene and superordinate schemas are grounded in fundamentally different ways: scene identity is encoded in image tokens throughout the network, while superordinate information emerges only late or not at all. Together, these results reveal that the organization of contextual inference in VLMs is more complex than accuracy alone suggests, with behavioral and mechanistic signatures

Related papers

• Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency [3.9792042887828565]
  We introduce Counterfactual Semantic Saliency (CSS) to measure the importance of objects.<n>To evaluate AI-human semantic alignment, we tested prominent vision-language models against a human psychophysics baseline.<n>Models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias) and high saliency objects.
  arXiv  Detail & Related papers  (2026-05-13T06:11:47Z)
• COOCO -- Common Objects Out-of-Context -- Semantic Violation in Scenes: Investigating Multimodal Context in Referential Communication [3.829419993067886]
  We show that Vision-Language Models (VLMs) learn to rely on scene contexts in a similar way, when generating references to objects.<n>Our findings show that models leverage scene context adaptively, depending on both the semantic relatedness between object and scene and the level of noise.
  arXiv  Detail & Related papers  (2025-06-27T14:44:45Z)
• Common Inpainted Objects In-N-Out of Context [21.387506141979188]
  Common Inpainted Objects In-N-Out of Context (COinCO) is a novel dataset addressing the scarcity of out-of-context examples in existing vision datasets.<n>By systematically replacing objects in COCO images through diffusion-based inpainting, we create 97,722 unique images featuring both contextually coherent and inconsistent scenes.
  arXiv  Detail & Related papers  (2025-05-31T21:42:12Z)
• Aligning and Prompting Everything All at Once for Universal Visual Perception [79.96124061108728]
  APE is a universal visual perception model for aligning and prompting everything all at once in an image to perform diverse tasks. APE advances the convergence of detection and grounding by reformulating language-guided grounding as open-vocabulary detection. Experiments on over 160 datasets demonstrate that APE outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-04T18:59:50Z)
• Position-Aware Contrastive Alignment for Referring Image Segmentation [65.16214741785633]
  We present a position-aware contrastive alignment network (PCAN) to enhance the alignment of multi-modal features. Our PCAN consists of two modules: 1) Position Aware Module (PAM), which provides position information of all objects related to natural language descriptions, and 2) Contrastive Language Understanding Module (CLUM), which enhances multi-modal alignment.
  arXiv  Detail & Related papers  (2022-12-27T09:13:19Z)
• Suspected Object Matters: Rethinking Model's Prediction for One-stage Visual Grounding [93.82542533426766]
  We propose a Suspected Object Transformation mechanism (SOT) to encourage the target object selection among the suspected ones. SOT can be seamlessly integrated into existing CNN and Transformer-based one-stage visual grounders. Extensive experiments demonstrate the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2022-03-10T06:41:07Z)
• Context-LGM: Leveraging Object-Context Relation for Context-Aware Object Recognition [48.5398871460388]
  We propose a novel Contextual Latent Generative Model (Context-LGM), which considers the object-context relation and models it in a hierarchical manner. To infer contextual features, we reformulate the objective function of Variational Auto-Encoder (VAE), where contextual features are learned as a posterior conditioned distribution on the object. The effectiveness of our method is verified by state-of-the-art performance on two context-aware object recognition tasks.
  arXiv  Detail & Related papers  (2021-10-08T11:31:58Z)
• Synthesizing the Unseen for Zero-shot Object Detection [72.38031440014463]
  We propose to synthesize visual features for unseen classes, so that the model learns both seen and unseen objects in the visual domain. We use a novel generative model that uses class-semantics to not only generate the features but also to discriminatively separate them.
  arXiv  Detail & Related papers  (2020-10-19T12:36:11Z)
• Deriving Visual Semantics from Spatial Context: An Adaptation of LSA and Word2Vec to generate Object and Scene Embeddings from Images [0.0]
  We develop two approaches for learning object and scene embeddings from annotated images. In the first approach, we generate embeddings from object co-occurrences in whole images, one for objects and one for scenes. In the second approach, rather than analyzing whole images of scenes, we focus on co-occurrences of objects within subregions of an image.
  arXiv  Detail & Related papers  (2020-09-20T08:26:38Z)
• COBE: Contextualized Object Embeddings from Narrated Instructional Video [52.73710465010274]
  We propose a new framework for learning Contextualized OBject Embeddings from automatically-transcribed narrations of instructional videos. We leverage the semantic and compositional structure of language by training a visual detector to predict a contextualized word embedding of the object and its associated narration. Our experiments show that our detector learns to predict a rich variety of contextual object information, and that it is highly effective in the settings of few-shot and zero-shot learning.
  arXiv  Detail & Related papers  (2020-07-14T19:04:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.