Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

• URL: http://arxiv.org/abs/2103.00868v1
• Date: Mon, 1 Mar 2021 09:37:27 GMT
• Title: Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning
• Authors: Alexander Jaus, Kailun Yang, Rainer Stiefelhagen
• Abstract summary: We introduce panoramic panoptic segmentation as the most holistic scene understanding. A complete surrounding understanding provides a maximum of information to the agent. We propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain.
• Score: 97.37544023666833
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we introduce panoramic panoptic segmentation as the most holistic scene understanding both in terms of field of view and image level understanding. A complete surrounding understanding provides a maximum of information to the agent, which is essential for any intelligent vehicle in order to make informed decisions in a safety-critical dynamic environment such as real-world traffic. In order to overcome the lack of annotated panoramic images, we propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain. Using our proposed method, we manage to achieve significant improvements of over 5\% measured in PQ over non-adapted models on our Wild Panoramic Panoptic Segmentation (WildPPS) dataset. We show that our proposed Panoramic Robust Feature (PRF) framework is not only suitable to improve performance on panoramic images but can be beneficial whenever model training and deployment are executed on data taken from different distributions. As an additional contribution, we publish WildPPS: The first panoramic panoptic image dataset to foster progress in surrounding perception.

Related papers

• Few-Shot Panoptic Segmentation With Foundation Models [23.231014713335664]
  We propose to leverage task-agnostic image features to enable few-shot panoptic segmentation by presenting Segmenting Panoptic Information with Nearly 0 labels (SPINO) In detail, our method combines a DINOv2 backbone with lightweight network heads for semantic segmentation and boundary estimation. We show that our approach, albeit being trained with only ten annotated images, predicts high-quality pseudo-labels that can be used with any existing panoptic segmentation method.
  arXiv  Detail & Related papers  (2023-09-19T16:09:01Z)
• Rethinking Range View Representation for LiDAR Segmentation [66.73116059734788]
  "Many-to-one" mapping, semantic incoherence, and shape deformation are possible impediments against effective learning from range view projections. We present RangeFormer, a full-cycle framework comprising novel designs across network architecture, data augmentation, and post-processing. We show that, for the first time, a range view method is able to surpass the point, voxel, and multi-view fusion counterparts in the competing LiDAR semantic and panoptic segmentation benchmarks.
  arXiv  Detail & Related papers  (2023-03-09T16:13:27Z)
• PanoViT: Vision Transformer for Room Layout Estimation from a Single Panoramic Image [11.053777620735175]
  PanoViT is a panorama vision transformer to estimate the room layout from a single panoramic image. Compared to CNN models, our PanoViT is more proficient in learning global information from the panoramic image. Our method outperforms state-of-the-art solutions in room layout prediction accuracy.
  arXiv  Detail & Related papers  (2022-12-23T05:37:11Z)
• Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation [73.48323921632506]
  We address panoramic semantic segmentation which is under-explored due to two critical challenges. First, we propose an upgraded Transformer for Panoramic Semantic, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable (DMLPv2) modules. Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images
  arXiv  Detail & Related papers  (2022-07-25T00:42:38Z)
• Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning [93.6645991946674]
  We introduce panoramic panoptic segmentation, as the most holistic scene understanding. A complete surrounding understanding provides a maximum of information to a mobile agent. We propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain.
  arXiv  Detail & Related papers  (2022-06-21T20:07:15Z)
• Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation [26.09267582056609]
  Large quantities of expensive, pixel-wise annotations are crucial for success of robust panoramic segmentation models. Distortions and the distinct image-feature distribution in 360-degree panoramas impede the transfer from the annotation-rich pinhole domain. We learn object deformations and panoramic image distortions in Deformable Patch Embedding (DPE) and Deformable Deformable (DMLP) components. Finally, we tie together shared semantics in pinhole- and panoramic feature embeddings by generating multi-scale prototype features.
  arXiv  Detail & Related papers  (2022-03-02T23:00:32Z)
• PANet: Perspective-Aware Network with Dynamic Receptive Fields and Self-Distilling Supervision for Crowd Counting [63.84828478688975]
  We propose a novel perspective-aware approach called PANet to address the perspective problem. Based on the observation that the size of the objects varies greatly in one image due to the perspective effect, we propose the dynamic receptive fields (DRF) framework. The framework is able to adjust the receptive field by the dilated convolution parameters according to the input image, which helps the model to extract more discriminative features for each local region.
  arXiv  Detail & Related papers  (2021-10-31T04:43:05Z)
• Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation [30.104947024614127]
  We formalize the task of unsupervised domain adaptation for panoramic semantic segmentation. DensePASS is a novel dataset for panoramic segmentation under cross-domain conditions. We introduce P2PDA - a generic framework for Pinhole-to-Panoramic semantic segmentation.
  arXiv  Detail & Related papers  (2021-10-21T11:22:05Z)
• DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization [66.25948693095604]
  We propose a novel method for panoramic 3D scene understanding which recovers the 3D room layout and the shape, pose, position, and semantic category for each object from a single full-view panorama image. Experiments demonstrate that our method outperforms existing methods on panoramic scene understanding in terms of both geometry accuracy and object arrangement.
  arXiv  Detail & Related papers  (2021-08-24T13:55:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.