Vision language models have difficulty recognizing virtual objects

• URL: http://arxiv.org/abs/2505.10453v1
• Date: Thu, 15 May 2025 16:11:33 GMT
• Title: Vision language models have difficulty recognizing virtual objects
• Authors: Tyler Tran, Sangeet Khemlani, J. G. Trafton,
• Abstract summary: Vision language models (VLMs) are AI systems paired with both language and vision encoders to process multimodal input.<n>We argue that descriptions of virtual objects can help test scene comprehension in these AI systems.
• Score: 0.20482269513546453
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision language models (VLMs) are AI systems paired with both language and vision encoders to process multimodal input. They are capable of performing complex semantic tasks such as automatic captioning, but it remains an open question about how well they comprehend the visuospatial properties of scenes depicted in the images they process. We argue that descriptions of virtual objects -- objects that are not visually represented in an image -- can help test scene comprehension in these AI systems. For example, an image that depicts a person standing under a tree can be paired with the following prompt: imagine that a kite is stuck in the tree. VLMs that comprehend the scene should update their representations and reason sensibly about the spatial relations between all three objects. We describe systematic evaluations of state-of-the-art VLMs and show that their ability to process virtual objects is inadequate.

Related papers

• Fake or Real, Can Robots Tell? Evaluating Embodied Vision-Language Models on Real and 3D-Printed Objects [3.9825600707172986]
  We present a comparative study of captioning strategies for tabletop scenes captured by a robotic arm equipped with an RGB camera.<n>The robot collects images of objects from multiple viewpoints, and we evaluate several models that generate scene descriptions.<n>Our experiments examine the trade-offs between single-view and multi-view captioning, and difference between recognising real-world and 3D printed objects.
  arXiv  Detail & Related papers  (2025-06-24T12:45:09Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• Help Me Identify: Is an LLM+VQA System All We Need to Identify Visual Concepts? [62.984473889987605]
  We present a zero-shot framework for fine-grained visual concept learning by leveraging large language model and Visual Question Answering (VQA) system. We pose these questions along with the query image to a VQA system and aggregate the answers to determine the presence or absence of an object in the test images. Our experiments demonstrate comparable performance with existing zero-shot visual classification methods and few-shot concept learning approaches.
  arXiv  Detail & Related papers  (2024-10-17T15:16:10Z)
• IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model [52.697180472760635]
  This paper explores the potential of character identities memory and recognition across multiple visual scenarios. We propose visual instruction tuning with ID reference and develop an ID-Aware Large Vision-Language Model, IDA-VLM. Our research introduces a novel benchmark MM-ID, to examine LVLMs on instance IDs memory and recognition across four dimensions.
  arXiv  Detail & Related papers  (2024-07-10T12:11:59Z)
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension [71.03445074045092]
  We propose ClawMachine, offering a new methodology that explicitly notates each entity using token collectives groups of visual tokens.<n>Our method unifies the prompt and answer of visual referential tasks without using additional syntax.<n>ClawMachine achieves superior performance on scene-level and referential understanding tasks with higher efficiency.
  arXiv  Detail & Related papers  (2024-06-17T08:39:16Z)
• Understanding Figurative Meaning through Explainable Visual Entailment [24.831452159672857]
  We propose a new task framing the figurative meaning understanding problem as an explainable visual entailment task.<n>We build the accompanying dataset V-FLUTE, containing 6,027 image, caption, label, explanation instances.<n>We find that VLMs struggle to generalize from literal to figurative meaning, particularly when it is present in images.
  arXiv  Detail & Related papers  (2024-05-02T17:07:25Z)
• Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects [11.117055725415446]
  Large Vision Language Models (LVLMs) have demonstrated impressive zero-shot capabilities in various vision-language dialogue scenarios. The absence of fine-grained visual object detection hinders the model from understanding the details of images, leading to irreparable visual hallucinations and factual errors. We propose Lyrics, a novel multi-modal pre-training and instruction fine-tuning paradigm that bootstraps vision-language alignment from fine-grained cross-modal collaboration.
  arXiv  Detail & Related papers  (2023-12-08T09:02:45Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)
• DeViL: Decoding Vision features into Language [53.88202366696955]
  Post-hoc explanation methods have often been criticised for abstracting away the decision-making process of deep neural networks. In this work, we would like to provide natural language descriptions for what different layers of a vision backbone have learned. We train a transformer network to translate individual image features of any vision layer into a prompt that a separate off-the-shelf language model decodes into natural language.
  arXiv  Detail & Related papers  (2023-09-04T13:59:55Z)
• Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions [4.026600887656479]
  We investigate whether a state-of-the-art language and vision model, CLIP, is able to ground perspective descriptions of a 3D object. We present an evaluation framework that uses a circling camera around a 3D object to generate images from different viewpoints. We find that a pre-trained CLIP model performs poorly on most canonical views.
  arXiv  Detail & Related papers  (2023-02-13T15:18:27Z)
• Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions [3.7957452405531256]
  This paper explores the potential of a state-of-the-art Vision and Language model, VinVL, to caption images at the scene level. We show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.
  arXiv  Detail & Related papers  (2022-11-09T15:33:51Z)
• ImaginaryNet: Learning Object Detectors without Real Images and Annotations [66.30908705345973]
  We propose a framework to synthesize images by combining pretrained language model and text-to-image model. With the synthesized images and class labels, weakly supervised object detection can then be leveraged to accomplish Imaginary-Supervised Object Detection. Experiments show that ImaginaryNet can (i) obtain about 70% performance in ISOD compared with the weakly supervised counterpart of the same backbone trained on real data.
  arXiv  Detail & Related papers  (2022-10-13T10:25:22Z)
• Language Grounding with 3D Objects [60.67796160959387]
  We introduce a novel reasoning task that targets both visual and non-visual language about 3D objects. We introduce several CLIP-based models for distinguishing objects. We find that adding view estimation to language grounding models improves accuracy on both SNARE and when identifying objects referred to in language on a robot platform.
  arXiv  Detail & Related papers  (2021-07-26T23:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.