Advancing Complex Wide-Area Scene Understanding with Hierarchical Coresets Selection

• URL: http://arxiv.org/abs/2507.13061v1
• Date: Thu, 17 Jul 2025 12:29:06 GMT
• Title: Advancing Complex Wide-Area Scene Understanding with Hierarchical Coresets Selection
• Authors: Jingyao Wang, Yiming Chen, Lingyu Si, Changwen Zheng,
• Abstract summary: Scene understanding is one of the core tasks in computer vision, aiming to extract semantic information from images to identify objects.<n>Existing Vision-Language Models (VLMs) face challenges in adaptation to unseen complex wide-area scenes.<n>This paper proposes a Hierarchical Coresets Selection mechanism to advance the adaptation ofVLMs in complex wide-area scene understanding.
• Score: 10.810165156142563
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scene understanding is one of the core tasks in computer vision, aiming to extract semantic information from images to identify objects, scene categories, and their interrelationships. Although advancements in Vision-Language Models (VLMs) have driven progress in this field, existing VLMs still face challenges in adaptation to unseen complex wide-area scenes. To address the challenges, this paper proposes a Hierarchical Coresets Selection (HCS) mechanism to advance the adaptation of VLMs in complex wide-area scene understanding. It progressively refines the selected regions based on the proposed theoretically guaranteed importance function, which considers utility, representativeness, robustness, and synergy. Without requiring additional fine-tuning, HCS enables VLMs to achieve rapid understandings of unseen scenes at any scale using minimal interpretable regions while mitigating insufficient feature density. HCS is a plug-and-play method that is compatible with any VLM. Experiments demonstrate that HCS achieves superior performance and universality in various tasks.

Related papers

• SeC: Advancing Complex Video Object Segmentation via Progressive Concept Construction [65.15449703659772]
  Video Object (VOS) is a core task in computer vision, requiring models to track and segment target objects across video frames.<n>We propose Segment Concept (SeC), a concept-driven segmentation framework that shifts from conventional feature matching to the progressive construction and utilization of high-level, object-centric representations.<n>SeC achieves an 11.8-point improvement over SAM SeCVOS, establishing a new state-of-the-art concept-aware video object segmentation.
  arXiv  Detail & Related papers  (2025-07-21T17:59:02Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)
• EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks [24.41705039390567]
  EmbodiedVSR (Embodied Visual Spatial Reasoning) is a novel framework that integrates dynamic scene graph-guided Chain-of-Thought (CoT) reasoning.<n>Our method enables zero-shot spatial reasoning without task-specific fine-tuning.<n>Experiments demonstrate that our framework significantly outperforms existing MLLM-based methods in accuracy and reasoning coherence.
  arXiv  Detail & Related papers  (2025-03-14T05:06:07Z)
• Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering [10.505845766495128]
  Multimodal large language models (MLLMs) have made significant progress in integrating visual and textual modalities.<n>We propose a novel framework based on multimodal retrieval-augmented generation (RAG)<n>RAG introduces structured scene graphs to enhance object recognition, relationship identification, and spatial understanding within images.
  arXiv  Detail & Related papers  (2024-12-30T13:16:08Z)
• Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models [50.98559225639266]
  We investigate the contributions of visual features from different encoder layers using 18 benchmarks spanning 6 task categories.<n>Our findings reveal that multilayer features provide complementary strengths with varying task dependencies, and uniform fusion leads to suboptimal performance.<n>We propose the instruction-guided vision aggregator, a module that dynamically integrates multi-layer visual features based on textual instructions.
  arXiv  Detail & Related papers  (2024-12-26T05:41:31Z)
• LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences [70.0873383646651]
  LSceneLLM is an adaptive framework that automatically identifies task-relevant areas.<n>A dense token selector examines the attention map of LLM to identify visual preferences for the instruction input.<n>An adaptive self-attention module is leveraged to fuse the coarse-grained and selected fine-grained visual information.
  arXiv  Detail & Related papers  (2024-12-02T09:07:57Z)
• HyperSeg: Towards Universal Visual Segmentation with Large Language Model [40.326267496697845]
  This paper aims to address universal segmentation for image and video perception with the strong reasoning ability empowered by Visual Large Language Models (VLLMs)<n>We propose HyperSeg, the first VLLM-based universal segmentation model for pixel-level image and video perception.
  arXiv  Detail & Related papers  (2024-11-26T17:18:20Z)
• FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity [68.15983300711355]
  FineCAPTION is a novel VLM that can recognize arbitrary masks as referential inputs and process high-resolution images for compositional image captioning at different levels. We introduce COMPOSITIONCAP, a new dataset for multi-grained region compositional image captioning, which introduces the task of compositional attribute-aware regional image captioning.
  arXiv  Detail & Related papers  (2024-11-23T02:20:32Z)
• Revisiting the Integration of Convolution and Attention for Vision Backbone [59.50256661158862]
  Convolutions and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. We propose in this work to use MSHAs and Convs in parallel textbfat different granularity levels instead. We empirically verify the potential of the proposed integration scheme, named textitGLMix: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few semantic slots.
  arXiv  Detail & Related papers  (2024-11-21T18:59:08Z)
• RegionGPT: Towards Region Understanding Vision Language Model [88.42271128373191]
  RegionGPT (short as RGPT) is a novel framework designed for complex region-level captioning and understanding. We develop an automated region caption data generation pipeline, enriching the training set with detailed region-level captions. We demonstrate that a universal RGPT model can be effectively applied and significantly enhancing performance across a range of region-level tasks.
  arXiv  Detail & Related papers  (2024-03-04T18:58:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.