How Can Objects Help Video-Language Understanding?

• URL: http://arxiv.org/abs/2504.07454v2
• Date: Tue, 05 Aug 2025 06:48:08 GMT
• Title: How Can Objects Help Video-Language Understanding?
• Authors: Zitian Tang, Shijie Wang, Junho Cho, Jaewook Yoo, Chen Sun,
• Abstract summary: We introduce ObjectML, a framework capable of leveraging arbitrary computer vision algorithm to extract and structured visual representation.<n>Through extensive evaluations on six video question benchmarks, we confirm that explicit integration of object-centric representation remains necessary.<n>Surprisingly, we observe that the simple approach quantizing the continuous, structured object information and representing them as plain text performs the best.
• Score: 16.63183488540909
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Do we still need to represent objects explicitly in multimodal large language models (MLLMs)? To one extreme, pre-trained encoders convert images into visual tokens, with which objects and spatiotemporal relationships may be implicitly modeled. To the other extreme, image captions by themselves provide strong empirical performances for understanding tasks, despite missing fine-grained spatiotemporal information. To answer this question, we introduce ObjectMLLM, a framework capable of leveraging arbitrary computer vision algorithm to extract and integrate structured visual representation. Through extensive evaluations on six video question answering benchmarks, we confirm that explicit integration of object-centric representation remains necessary. Surprisingly, we observe that the simple approach of quantizing the continuous, structured object information and representing them as plain text performs the best, offering a data-efficient approach to integrate other visual perception modules into MLLM design. Our code and models are released at https://github.com/brown-palm/ObjectMLLM.

Related papers

• Re-Aligning Language to Visual Objects with an Agentic Workflow [73.73778652260911]
  Language-based object detection aims to align visual objects with language expressions.<n>Recent studies leverage vision-language models (VLMs) to automatically generate human-like expressions for visual objects.<n>We propose an agentic workflow controlled by an LLM to re-align language to visual objects via adaptively adjusting image and text prompts.
  arXiv  Detail & Related papers  (2025-03-30T16:41:12Z)
• EagleVision: Object-level Attribute Multimodal LLM for Remote Sensing [3.3072144045024396]
  EagleVision is an MLLM tailored for remote sensing that excels in object detection and attribute comprehension.<n>We construct EVAttrs-95K, the first large-scale object attribute understanding dataset in RS for instruction tuning.<n>EagleVision achieves state-of-the-art performance on both fine-grained object detection and object attribute understanding tasks.
  arXiv  Detail & Related papers  (2025-03-30T06:13:13Z)
• Where do Large Vision-Language Models Look at when Answering Questions? [35.39354978511109]
  Large Vision-Language Models (LVLMs) have shown promising performance in vision-language understanding and reasoning tasks. We extend existing heatmap visualization methods to support LVLMs for open-ended visual question answering. We conduct a comprehensive analysis of state-of-the-art LVLMs on benchmarks designed to require visual information to answer.
  arXiv  Detail & Related papers  (2025-03-18T04:34:43Z)
• Help Me Identify: Is an LLM+VQA System All We Need to Identify Visual Concepts? [62.984473889987605]
  We present a zero-shot framework for fine-grained visual concept learning by leveraging large language model and Visual Question Answering (VQA) system. We pose these questions along with the query image to a VQA system and aggregate the answers to determine the presence or absence of an object in the test images. Our experiments demonstrate comparable performance with existing zero-shot visual classification methods and few-shot concept learning approaches.
  arXiv  Detail & Related papers  (2024-10-17T15:16:10Z)
• Towards Interpreting Visual Information Processing in Vision-Language Models [24.51408101801313]
  Vision-Language Models (VLMs) are powerful tools for processing and understanding text and images.<n>We study the processing of visual tokens in the language model component of LLaVA, a prominent VLM.
  arXiv  Detail & Related papers  (2024-10-09T17:55:02Z)
• VisMin: Visual Minimal-Change Understanding [7.226130826257802]
  We introduce a new, challenging benchmark termed Visual Minimal-Change Understanding (VisMin)<n>VisMin requires models to predict the correct image-caption match given two images and two captions.<n>We build an automatic framework using large language models and diffusion models, followed by a rigorous 4-step verification process by human annotators.
  arXiv  Detail & Related papers  (2024-07-23T18:10:43Z)
• Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge [76.45868419402265]
  multimodal large language models (MLLMs) have made significant strides by training on vast high-quality image-text datasets. However, the inherent difficulty in explicitly conveying fine-grained or spatially dense information in text, such as masks, poses a challenge for MLLMs. This paper proposes a new visual prompt approach to integrate fine-grained external knowledge, gleaned from specialized vision models, into MLLMs.
  arXiv  Detail & Related papers  (2024-07-05T17:43:30Z)
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension [71.03445074045092]
  We propose ClawMachine, offering a new methodology that explicitly notates each entity using token collectives groups of visual tokens.<n>Our method unifies the prompt and answer of visual referential tasks without using additional syntax.<n>ClawMachine achieves superior performance on scene-level and referential understanding tasks with higher efficiency.
  arXiv  Detail & Related papers  (2024-06-17T08:39:16Z)
• Towards Semantic Equivalence of Tokenization in Multimodal LLM [149.11720372278273]
  Vision tokenization is essential for semantic alignment between vision and language.<n>This paper proposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok)<n>SeTok groups visual features into semantic units via a dynamic clustering algorithm.<n>The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features.
  arXiv  Detail & Related papers  (2024-06-07T17:55:43Z)
• 1st Place Solution for MOSE Track in CVPR 2024 PVUW Workshop: Complex Video Object Segmentation [72.54357831350762]
  We propose a semantic embedding video object segmentation model and use the salient features of objects as query representations. We trained our model on a large-scale video object segmentation dataset. Our model achieves first place (textbf84.45%) in the test set of Complex Video Object Challenge.
  arXiv  Detail & Related papers  (2024-06-07T03:13:46Z)
• OLIVE: Object Level In-Context Visual Embeddings [8.168219870640318]
  We propose a novel method to prompt large language models with in-context visual object vectors. This eliminates the necessity of fusing a lengthy array of image patch features and significantly speeds up training. Our experiments reveal that our method achieves competitive referring object classification and captioning performance.
  arXiv  Detail & Related papers  (2024-06-02T21:36:31Z)
• Auto-Encoding Morph-Tokens for Multimodal LLM [151.2618346912529]
  We propose encoding images into morph-tokens to serve a dual purpose: for comprehension, they act as visual prompts instructing MLLM to generate texts. Experiments show that morph-tokens can achieve a new SOTA for multimodal comprehension and generation simultaneously.
  arXiv  Detail & Related papers  (2024-05-03T08:43:06Z)
• OSCaR: Object State Captioning and State Change Representation [52.13461424520107]
  This paper introduces the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark. OSCaR consists of 14,084 annotated video segments with nearly 1,000 unique objects from various egocentric video collections. It sets a new testbed for evaluating multimodal large language models (MLLMs)
  arXiv  Detail & Related papers  (2024-02-27T01:48:19Z)
• CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models [58.95889895912716]
  We introduce a new benchmark, named as CODIS, designed to assess the ability of models to use context provided in free-form text to enhance visual comprehension. Our findings indicate that MLLMs consistently fall short of human performance on this benchmark. This underscores the pressing need to enhance the ability of MLLMs to comprehend visuals in a context-dependent manner.
  arXiv  Detail & Related papers  (2024-02-21T08:21:12Z)
• Leveraging VLM-Based Pipelines to Annotate 3D Objects [68.51034848207355]
  We propose an alternative algorithm to marginalize over factors such as the viewpoint that affect the VLM's response. Instead of merging text-only responses, we utilize the VLM's joint image-text likelihoods. We show how a VLM-based pipeline can be leveraged to produce reliable annotations for 764K objects from the 764K dataset.
  arXiv  Detail & Related papers  (2023-11-29T17:54:22Z)
• Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association [23.06058982328083]
  We focus on the wide existence of reporting bias in visual-language datasets. We propose a bimodal augmentation (BiAug) approach to mitigate this bias. BiAug synthesizes visual-language examples with a rich array of object-attribute pairing and construct cross-modal hard negatives.
  arXiv  Detail & Related papers  (2023-10-02T16:48:50Z)
• Object-Centric Representation Learning for Video Question Answering [27.979053252431306]
  Video answering (Video QA) presents a powerful testbed for human-like intelligent behaviors. The task demands new capabilities to integrate processing, language understanding, binding abstract concepts to concrete visual artifacts. We propose a new query-guided representation framework to turn a video into a relational graph of objects.
  arXiv  Detail & Related papers  (2021-04-12T02:37:20Z)
• ClawCraneNet: Leveraging Object-level Relation for Text-based Video Segmentation [47.7867284770227]
  Text-based video segmentation is a challenging task that segments out the natural language referred objects in videos. We introduce a novel top-down approach by imitating how we human segment an object with the language guidance. Our method outperforms state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2021-03-19T09:31:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.