With Great Context Comes Great Prediction Power: Classifying Objects via Geo-Semantic Scene Graphs

• URL: http://arxiv.org/abs/2512.23024v1
• Date: Sun, 28 Dec 2025 17:53:55 GMT
• Title: With Great Context Comes Great Prediction Power: Classifying Objects via Geo-Semantic Scene Graphs
• Abstract summary: This paper argues for the critical role of context and introduces a novel framework for contextual object classification.<n>We first construct a Geo-Semantic Contextual Graph (GSCG) from a single monocular image.<n>This explicit graph structure makes the model's reasoning process inherently interpretable.
• Score: 5.492064811668243
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Humans effortlessly identify objects by leveraging a rich understanding of the surrounding scene, including spatial relationships, material properties, and the co-occurrence of other objects. In contrast, most computational object recognition systems operate on isolated image regions, devoid of meaning in isolation, thus ignoring this vital contextual information. This paper argues for the critical role of context and introduces a novel framework for contextual object classification. We first construct a Geo-Semantic Contextual Graph (GSCG) from a single monocular image. This rich, structured representation is built by integrating a metric depth estimator with a unified panoptic and material segmentation model. The GSCG encodes objects as nodes with detailed geometric, chromatic, and material attributes, and their spatial relationships as edges. This explicit graph structure makes the model's reasoning process inherently interpretable. We then propose a specialized graph-based classifier that aggregates features from a target object, its immediate neighbors, and the global scene context to predict its class. Through extensive ablation studies, we demonstrate that our context-aware model achieves a classification accuracy of 73.4%, dramatically outperforming context-agnostic versions (as low as 38.4%). Furthermore, our GSCG-based approach significantly surpasses strong baselines, including fine-tuned ResNet models (max 53.5%) and a state-of-the-art multimodal Large Language Model (LLM), Llama 4 Scout, which, even when given the full image alongside a detailed description of objects, maxes out at 42.3%. These results on COCO 2017 train/val splits highlight the superiority of explicitly structured and interpretable context for object recognition tasks.

Related papers

• KeySG: Hierarchical Keyframe-Based 3D Scene Graphs [1.5134439544218246]
  KeySG represents 3D scenes as a hierarchical graph consisting of floors, rooms, objects, and functional elements.<n>We leverage VLM to extract scene information, alleviating the need to explicitly model relationship edges between objects.<n>Our approach can process complex and ambiguous queries while mitigating the scalability issues associated with large scene graphs.
  arXiv  Detail & Related papers  (2025-10-01T15:53:27Z)
• Synthetic Visual Genome [88.00433979509218]
  We introduce ROBIN: an instruction-tuned with densely annotated relationships capable of constructing high-quality dense graphs at scale.<n>In total, our dataset contains 146K images and 5.6M relationships for 2.6M objects.<n> ROBIN-3B model, despite being trained on less than 3 million instances, outperforms similar-size models trained on over 300 million instances on relationship understanding benchmarks.
  arXiv  Detail & Related papers  (2025-06-09T11:09:10Z)
• GraPLUS: Graph-based Placement Using Semantics for Image Composition [3.0450307343472405]
  We present GraPLUS (Graph-based Placement Using Semantics), a novel framework for plausible object placement in images.<n>Our approach uniquely combines graph-structured scene representation with semantic understanding to determine contextually appropriate object positions.<n>GraPLUS placement accuracy of 92.1% and an FID score of 28.83 on the OPA dataset, outperforming state-of-the-art methods by 8.1% while maintaining competitive visual quality.
  arXiv  Detail & Related papers  (2025-03-20T00:43:29Z)
• Open-Vocabulary Octree-Graph for 3D Scene Understanding [54.11828083068082]
  Octree-Graph is a novel scene representation for open-vocabulary 3D scene understanding. An adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape.
  arXiv  Detail & Related papers  (2024-11-25T10:14:10Z)
• On Support Relations Inference and Scene Hierarchy Graph Construction from Point Cloud in Clustered Environments [3.4535508414601344]
  In 3D scenes, rich spatial geometric and topological information are often ignored by RGB-based approaches for scene understanding. In this study, we develop a bottom-up approach for scene understanding that infers support relations between objects from a point cloud.
  arXiv  Detail & Related papers  (2024-04-22T02:42:32Z)
• Less is More: Toward Zero-Shot Local Scene Graph Generation via Foundation Models [16.08214739525615]
  We present a new task called Local Scene Graph Generation. It aims to abstract pertinent structural information with partial objects and their relationships in an image. We introduce zEro-shot Local scEne GrAph geNeraTion (ELEGANT), a framework harnessing foundation models renowned for their powerful perception and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-10-02T17:19:04Z)
• Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models [3.86170450233149]
  We show that large vision-and-language models (VLMs) trained to match images with text lack fine-grained understanding of spatial relations. We propose an alternative fine-grained, compositional approach for recognizing and ranking spatial clauses.
  arXiv  Detail & Related papers  (2023-08-18T18:58:54Z)
• Relation Regularized Scene Graph Generation [206.76762860019065]
  Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations. We propose a relation regularized network (R2-Net) which can predict whether there is a relationship between two objects. Our R2-Net can effectively refine object labels and generate scene graphs.
  arXiv  Detail & Related papers  (2022-02-22T11:36:49Z)
• Unsupervised Part Discovery from Contrastive Reconstruction [90.88501867321573]
  The goal of self-supervised visual representation learning is to learn strong, transferable image representations. We propose an unsupervised approach to object part discovery and segmentation. Our method yields semantic parts consistent across fine-grained but visually distinct categories.
  arXiv  Detail & Related papers  (2021-11-11T17:59:42Z)
• ORD: Object Relationship Discovery for Visual Dialogue Generation [60.471670447176656]
  We propose an object relationship discovery (ORD) framework to preserve the object interactions for visual dialogue generation. A hierarchical graph convolutional network (HierGCN) is proposed to retain the object nodes and neighbour relationships locally, and then refines the object-object connections globally. Experiments have proved that the proposed method can significantly improve the quality of dialogue by utilising the contextual information of visual relationships.
  arXiv  Detail & Related papers  (2020-06-15T12:25:40Z)
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning [78.3857991931479]
  We present GROLLA, an evaluation framework for Grounded Language Learning with Attributes. We also propose a new dataset CompGuessWhat?! as an instance of this framework for evaluating the quality of learned neural representations.
  arXiv  Detail & Related papers  (2020-06-03T11:21:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.