SceneParser: Hierarchical Scene Parsing for Visual Semantics Understanding

• URL: http://arxiv.org/abs/2605.14923v1
• Date: Thu, 14 May 2026 14:58:46 GMT
• Title: SceneParser: Hierarchical Scene Parsing for Visual Semantics Understanding
• Abstract summary: General scene perception has progressed from object recognition toward open-vocabulary grounding, part localization, and affordance prediction.<n>These capabilities are often realized as isolated predictions that localize objects, parts, or interaction points without capturing the structured dependencies needed for interaction-oriented scene understanding.<n>We introduce Hierarchical Scene Parsing, an interaction-oriented parsing task that represents physical scenes as explicit scene -> object -> part -> affordance hierarchies with cross-level bindings.
• Score: 81.36825413112614
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: General scene perception has progressed from object recognition toward open-vocabulary grounding, part localization, and affordance prediction. Yet these capabilities are often realized as isolated predictions that localize objects, parts, or interaction points without capturing the structured dependencies needed for interaction-oriented scene understanding. To address this gap, we introduce Hierarchical Scene Parsing, an interaction-oriented parsing task that represents physical scenes as explicit scene -> object -> part -> affordance hierarchies with cross-level bindings. We instantiate this task with SceneParser, a VLM-based parser trained for unified hierarchical generation with structural-completion pseudo labels and curriculum learning. To support training and evaluation, we construct SceneParser-Bench, a large-scale benchmark built with a scalable hierarchical data engine, containing 110K training images, a 5K validation split, 777K objects, 1.14M parts, 1.74M affordance annotations, and 1.74M valid object-part-affordance chain instances. We further introduce Level-1 to Level-3 conditional metrics and ParseRate to evaluate localization, cross-level binding, and hierarchical completeness. Experiments show that existing MLLMs and perception-stitching pipelines struggle with hierarchical parsing on our SceneParser-Bench, while SceneParser achieves stronger structure-aware performance. Besides, ablations, evaluations on COCO and AGD20K, and a downstream planning probe demonstrate that our SceneParser is compatible with conventional tasks and provides an actionable representation for visual understanding.

Related papers

• Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation [18.782084217347187]
  Affordance2Action (A2A) is a benchmark-centered learning framework for scene-level, task-conditioned part affordance grounding.<n>A2A-Bench is a manipulation-oriented benchmark that covers both single-region and multi-region instruction correspondences in everyday scenes.<n>A2A exposes substantial gaps in generic segmentation, VLM-based grounding, and affordance distillation baselines.
  arXiv  Detail & Related papers  (2026-06-02T19:36:13Z)
• SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models [63.28435103335999]
  We introduce SOCO, a new benchmark for Semantic Object Correspondence.<n>We show that vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories.<n>We also show that LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching.
  arXiv  Detail & Related papers  (2026-05-29T17:58:48Z)
• SceneForge: Structured World Supervision from 3D Interventions [5.973748478214713]
  multimodal learning tasks require supervision that remains consistent across edits, viewpoints, and scene-level interventions.<n>We present SceneForge, an intervention-driven framework that generates structured supervision from editable 3D world states.
  arXiv  Detail & Related papers  (2026-05-14T05:38:00Z)
• Contextual inference from single objects in Vision-Language models [10.367669666212473]
  How much scene context a single object carries is a well-studied question in human scene perception.<n>How this capacity is organized in vision-language models (VLMs) remains poorly understood.<n>We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects.
  arXiv  Detail & Related papers  (2026-03-20T13:24:15Z)
• Object-Centric Framework for Video Moment Retrieval [15.916994168542345]
  Most existing moment retrieval methods rely on temporal sequences of frame-level features that primarily encode global visual and semantic information.<n>In particular temporal dynamics at the object level have been largely overlooked, limiting existing approaches in scenarios requiring object-level reasoning.<n>Our method first extracts query-relevant objects using a scene graph and then graphs from video frames to represent these objects and their relationships.<n>Based on the scene graphs, we construct object-level feature sequences that encode rich visual and semantic information. These sequences are processed by a video tracklet transformer, which models relational-temporal localization among objects over time.
  arXiv  Detail & Related papers  (2025-12-20T17:44:53Z)
• LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation [56.12844551763724]
  We propose LangHOPS, the first Multimodal Large Language Model (MLLM) based framework for open-vocabulary object-part instance segmentation.<n>LangHOPS can jointly detect and segment hierarchical object and part instances from open-vocabulary candidate categories.
  arXiv  Detail & Related papers  (2025-10-29T08:21:59Z)
• ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splatting [54.92763171355442]
  ObjectGS is an object-aware framework that unifies 3D scene reconstruction with semantic understanding.<n>We show through experiments that ObjectGS outperforms state-of-the-art methods on open-vocabulary and panoptic segmentation tasks.
  arXiv  Detail & Related papers  (2025-07-21T10:06:23Z)
• Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning [77.2852342808769]
  In this paper, we introduce a detailed caption benchmark, termed as CompreCap, to evaluate the visual context from a directed scene graph view.<n>We first manually segment the image into semantically meaningful regions according to common-object vocabulary, while also distinguishing attributes of objects within all those regions.<n>Then directional relation labels of these objects are annotated to compose a directed scene graph that can well encode rich compositional information of the image.
  arXiv  Detail & Related papers  (2024-12-11T18:37:42Z)
• Barking Up The Syntactic Tree: Enhancing VLM Training with Syntactic Losses [31.85977999591524]
  Vision-Language Models implicitly learn to associate image regions with words from large-scale training data.<n>Rich semantic and syntactic structures within the text modality have been overlooked as sources of supervision.<n>Hierarchically STructured Learning (HIST) enhances spatial vision-language alignment without using additional human annotations.
  arXiv  Detail & Related papers  (2024-12-11T05:36:18Z)
• Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens [93.98605636451806]
  StructureViT shows how utilizing the structure of a small number of images only available during training can improve a video model. SViT shows strong performance improvements on multiple video understanding tasks and datasets.
  arXiv  Detail & Related papers  (2022-06-13T17:45:05Z)
• Boundary-aware Self-supervised Learning for Video Scene Segmentation [20.713635723315527]
  Video scene segmentation is a task of temporally localizing scene boundaries in a video. We introduce three novel boundary-aware pretext tasks: Shot-Scene Matching, Contextual Group Matching and Pseudo-boundary Prediction. We achieve the new state-of-the-art on the MovieNet-SSeg benchmark.
  arXiv  Detail & Related papers  (2022-01-14T02:14:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.