Learning Object State Changes in Videos: An Open-World Perspective

• URL: http://arxiv.org/abs/2312.11782v2
• Date: Wed, 3 Apr 2024 16:57:35 GMT
• Title: Learning Object State Changes in Videos: An Open-World Perspective
• Abstract summary: We introduce a novel open-world formulation for the video Object State Changes problem. The goal is to temporally localize the three stages of an OSC -- the object's initial state, its transitioning state, and its end state. We present HowToChange, the first open-world benchmark for video OSC localization.
• Score: 55.757614346710014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object State Changes (OSCs) are pivotal for video understanding. While humans can effortlessly generalize OSC understanding from familiar to unknown objects, current approaches are confined to a closed vocabulary. Addressing this gap, we introduce a novel open-world formulation for the video OSC problem. The goal is to temporally localize the three stages of an OSC -- the object's initial state, its transitioning state, and its end state -- whether or not the object has been observed during training. Towards this end, we develop VidOSC, a holistic learning approach that: (1) leverages text and vision-language models for supervisory signals to obviate manually labeling OSC training data, and (2) abstracts fine-grained shared state representations from objects to enhance generalization. Furthermore, we present HowToChange, the first open-world benchmark for video OSC localization, which offers an order of magnitude increase in the label space and annotation volume compared to the best existing benchmark. Experimental results demonstrate the efficacy of our approach, in both traditional closed-world and open-world scenarios.

Related papers

• OSCBench: Benchmarking Object State Change in Text-to-Video Generation [47.72341406051041]
  We introduce OSCBench, a benchmark specifically designed to assess object state change (OSC) performance in text-to-video generation models.<n>OSC refers to the transformation of an object's state induced by an action, such as peeling a potato or slicing a lemon.<n>We evaluate six representative open-source and proprietary T2V models using both human user study and multimodal large language model (MLLM)-based automatic evaluation.
  arXiv  Detail & Related papers  (2026-03-12T09:08:01Z)
• Learning Event Completeness for Weakly Supervised Video Anomaly Detection [5.140169437190526]
  We present a novel Learning Event Completeness for Weakly Supervised Video Anomaly Detection (LEC-VAD)<n>LEC-VAD encodes both category-aware and category-agnostic semantics between vision and language.<n>We develop a novel memory bank-based prototype learning mechanism to enrich concise text descriptions associated with anomaly-event categories.
  arXiv  Detail & Related papers  (2025-06-16T04:56:58Z)
• SPOC: Spatially-Progressing Object State Change Segmentation in Video [52.65373395382122]
  We introduce the spatially-progressing object state change segmentation task. The goal is to segment at the pixel-level those regions of an object that are actionable and those that are transformed. We demonstrate useful implications for tracking activity progress to benefit robotic agents.
  arXiv  Detail & Related papers  (2025-03-15T01:48:54Z)
• Hawk: Learning to Understand Open-World Video Anomalies [76.9631436818573]
  Video Anomaly Detection (VAD) systems can autonomously monitor and identify disturbances, reducing the need for manual labor and associated costs. We introduce Hawk, a novel framework that leverages interactive large Visual Language Models (VLM) to interpret video anomalies precisely. We have annotated over 8,000 anomaly videos with language descriptions, enabling effective training across diverse open-world scenarios, and also created 8,000 question-answering pairs for users' open-world questions.
  arXiv  Detail & Related papers  (2024-05-27T07:08:58Z)
• Anticipating Object State Changes [0.8428703116072809]
  The proposed framework predicts object state changes that will occur in the near future due to yet unseen human actions. It integrates learned visual features that represent recent visual information with natural language (NLP) features that represent past object state changes and actions. The proposed approach also underscores the potential of integrating video and linguistic cues to enhance the predictive performance of video understanding systems.
  arXiv  Detail & Related papers  (2024-05-21T13:40:30Z)
• Open-Vocabulary Spatio-Temporal Action Detection [59.91046192096296]
  Open-vocabulary-temporal action detection (OV-STAD) is an important fine-grained video understanding task. OV-STAD requires training a model on a limited set of base classes with box and label supervision. To better adapt the holistic VLM for the fine-grained action detection task, we carefully fine-tune it on the localized video region-text pairs.
  arXiv  Detail & Related papers  (2024-05-17T14:52:47Z)
• OSCaR: Object State Captioning and State Change Representation [52.13461424520107]
  This paper introduces the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark. OSCaR consists of 14,084 annotated video segments with nearly 1,000 unique objects from various egocentric video collections. It sets a new testbed for evaluating multimodal large language models (MLLMs)
  arXiv  Detail & Related papers  (2024-02-27T01:48:19Z)
• Unsupervised Open-Vocabulary Object Localization in Videos [118.32792460772332]
  We show that recent advances in video representation learning and pre-trained vision-language models allow for substantial improvements in self-supervised video object localization. We propose a method that first localizes objects in videos via an object-centric approach with slot attention and then assigns text to the obtained slots.
  arXiv  Detail & Related papers  (2023-09-18T15:20:13Z)
• Cross-Video Contextual Knowledge Exploration and Exploitation for Ambiguity Reduction in Weakly Supervised Temporal Action Localization [23.94629999419033]
  Weakly supervised temporal action localization (WSTAL) aims to localize actions in untrimmed videos using video-level labels. Our work addresses this from a novel perspective, by exploring and exploiting the cross-video contextual knowledge within the dataset. Our method outperforms the state-of-the-art methods, and can be easily plugged into other WSTAL methods.
  arXiv  Detail & Related papers  (2023-08-24T07:19:59Z)
• Open-World Weakly-Supervised Object Localization [26.531408294517416]
  We introduce a new weakly-supervised object localization task called OWSOL (Open-World Weakly-Supervised Object localization) We propose a novel paradigm of contrastive representation co-learning using both labeled and unlabeled data to generate a complete G-CAM for object localization. We re-organize two widely used datasets, i.e., ImageNet-1K and iNatLoc500, and propose OpenImages150 to serve as evaluation benchmarks for OWSOL.
  arXiv  Detail & Related papers  (2023-04-17T13:31:59Z)
• ActBERT: Learning Global-Local Video-Text Representations [74.29748531654474]
  We introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data. We leverage global action information to catalyze the mutual interactions between linguistic texts and local regional objects. ActBERT significantly outperforms the state-of-the-arts, demonstrating its superiority in video-text representation learning.
  arXiv  Detail & Related papers  (2020-11-14T07:14:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.