Multi-task Fusion for Efficient Panoptic-Part Segmentation

• URL: http://arxiv.org/abs/2212.07671v2
• Date: Mon, 19 Dec 2022 13:19:21 GMT
• Title: Multi-task Fusion for Efficient Panoptic-Part Segmentation
• Authors: Sravan Kumar Jagadeesh, Ren\'e Schuster, Didier Stricker
• Abstract summary: We introduce a novel network that generates semantic, instance, and part segmentation using a shared encoder. To fuse the predictions of all three heads efficiently, we introduce a parameter-free joint fusion module. Our method is evaluated on the Cityscapes Panoptic Parts ( CPP) and Pascal Panoptic Parts (PPP) datasets.
• Score: 12.650574326251023
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce a novel network that generates semantic, instance, and part segmentation using a shared encoder and effectively fuses them to achieve panoptic-part segmentation. Unifying these three segmentation problems allows for mutually improved and consistent representation learning. To fuse the predictions of all three heads efficiently, we introduce a parameter-free joint fusion module that dynamically balances the logits and fuses them to create panoptic-part segmentation. Our method is evaluated on the Cityscapes Panoptic Parts (CPP) and Pascal Panoptic Parts (PPP) datasets. For CPP, the PartPQ of our proposed model with joint fusion surpasses the previous state-of-the-art by 1.6 and 4.7 percentage points for all areas and segments with parts, respectively. On PPP, our joint fusion outperforms a model using the previous top-down merging strategy by 3.3 percentage points in PartPQ and 10.5 percentage points in PartPQ for partitionable classes.

Related papers

• PCF-Lift: Panoptic Lifting by Probabilistic Contrastive Fusion [80.79938369319152]
  We design a new pipeline coined PCF-Lift based on our Probabilis-tic Contrastive Fusion (PCF) Our PCF-lift not only significantly outperforms the state-of-the-art methods on widely used benchmarks including the ScanNet dataset and the Messy Room dataset (4.4% improvement of scene-level PQ)
  arXiv  Detail & Related papers  (2024-10-14T16:06:59Z)
• More than the Sum of Its Parts: Ensembling Backbone Networks for Few-Shot Segmentation [49.090592800481616]
  We study whether fusing features from different backbones can improve the ability of acrlongfss models to capture richer visual features. We propose and compare two ensembling techniques-Independent Voting and Feature Fusion. Our approach outperforms the original single-backbone PANet across standard benchmarks even in challenging one-shot learning scenarios.
  arXiv  Detail & Related papers  (2024-02-09T18:01:15Z)
• JPPF: Multi-task Fusion for Consistent Panoptic-Part Segmentation [12.19926973291957]
  Part-aware panoptic segmentation is a problem of computer vision that aims to provide a semantic understanding of the scene at multiple levels of granularity. We present our Joint Panoptic Part Fusion (JPPF) that combines the three individual segmentations effectively to obtain a panoptic-part segmentation.
  arXiv  Detail & Related papers  (2023-11-30T15:17:46Z)
• Compositor: Bottom-up Clustering and Compositing for Robust Part and Object Segmentation [16.48046112716597]
  We present a robust approach for joint part and object segmentation. We build a hierarchical feature representation including pixel, part, and object-level embeddings to solve it in a bottom-up manner. This bottom-up interaction is shown to be effective in integrating information from lower semantic levels to higher semantic levels.
  arXiv  Detail & Related papers  (2023-06-12T20:12:02Z)
• Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation [49.56131393810713]
  We present an SE(3) equivariant architecture and a training strategy to tackle this task in an unsupervised manner. Our method excels in both model performance and computational efficiency, with only 0.25M parameters and 0.92G FLOPs.
  arXiv  Detail & Related papers  (2023-06-08T22:55:32Z)
• PanopticPartFormer++: A Unified and Decoupled View for Panoptic Part Segmentation [153.76253697804225]
  Panoptic Part (PPS) unifies panoptic and part segmentation into one task. We design the first end-to-end unified framework, Panoptic-PartFormer. Our models can serve as a strong baseline and aid future research in PPS.
  arXiv  Detail & Related papers  (2023-01-03T05:30:56Z)
• Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation [63.910211095033596]
  Few-shot segmentation aims to segment unseen-class objects given only a handful of densely labeled samples. We propose a simple yet versatile framework in the spirit of divide-and-conquer. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information.
  arXiv  Detail & Related papers  (2022-04-21T06:21:14Z)
• Panoptic-PartFormer: Learning a Unified Model for Panoptic Part Segmentation [76.9420522112248]
  Panoptic Part (PPS) aims to unify panoptic segmentation and part segmentation into one task. We design the first end-to-end unified method named Panoptic-PartFormer. Our Panoptic-PartFormer achieves the new state-of-the-art results on both Cityscapes PPS and Pascal Context PPS datasets.
  arXiv  Detail & Related papers  (2022-04-10T11:16:45Z)
• Part-aware Panoptic Segmentation [3.342126234995932]
  Part-aware Panoptic (PPS) aims to understand a scene at multiple levels of abstraction. We provide consistent annotations on two commonly used datasets: Cityscapes and Pascal VOC. We present a single metric to evaluate PPS, called Part-aware Panoptic Quality (PartPQ)
  arXiv  Detail & Related papers  (2021-06-11T12:48:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.