Framework-agnostic Semantically-aware Global Reasoning for Segmentation

• URL: http://arxiv.org/abs/2212.03338v2
• Date: Wed, 17 Apr 2024 23:02:39 GMT
• Title: Framework-agnostic Semantically-aware Global Reasoning for Segmentation
• Authors: Mir Rayat Imtiaz Hossain, Leonid Sigal, James J. Little,
• Abstract summary: We propose a component that learns to project image features into latent representations and reason between them. Our design encourages the latent regions to represent semantic concepts by ensuring that the activated regions are spatially disjoint. Our latent tokens are semantically interpretable and diverse and provide a rich set of features that can be transferred to downstream tasks.
• Score: 29.69187816377079
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in pixel-level tasks (e.g. segmentation) illustrate the benefit of of long-range interactions between aggregated region-based representations that can enhance local features. However, such aggregated representations, often in the form of attention, fail to model the underlying semantics of the scene (e.g. individual objects and, by extension, their interactions). In this work, we address the issue by proposing a component that learns to project image features into latent representations and reason between them using a transformer encoder to generate contextualized and scene-consistent representations which are fused with original image features. Our design encourages the latent regions to represent semantic concepts by ensuring that the activated regions are spatially disjoint and the union of such regions corresponds to a connected object segment. The proposed semantic global reasoning (SGR) component is end-to-end trainable and can be easily added to a wide variety of backbones (CNN or transformer-based) and segmentation heads (per-pixel or mask classification) to consistently improve the segmentation results on different datasets. In addition, our latent tokens are semantically interpretable and diverse and provide a rich set of features that can be transferred to downstream tasks like object detection and segmentation, with improved performance. Furthermore, we also proposed metrics to quantify the semantics of latent tokens at both class \& instance level.

Related papers

• AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing [12.91626624625134]
  Few-shot (FSS) aims to segment the interested objects in the query image with just a handful of labeled samples (i.e., support images) Previous schemes would leverage the similarity between support-Query pixel pairs to construct the pixel-level semantic correlation. In remote sensing scenarios with extreme intra-class variations and cluttered backgrounds, such pixel-level correlations may produce tremendous mismatches. We propose a novel Agent Mining Transformer (AgMTR), which adaptively mines a set of local-aware agents to construct agent-level semantic correlation.
  arXiv  Detail & Related papers  (2024-09-26T01:12:01Z)
• EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation [5.476136494434766]
  We introduce EiCue, a technique providing semantic and structural cues through an eigenbasis derived from semantic similarity matrix. We guide our model to learn object-level representations with intra- and inter-image object-feature consistency. Experiments on COCO-Stuff, Cityscapes, and Potsdam-3 datasets demonstrate the state-of-the-art USS results.
  arXiv  Detail & Related papers  (2024-03-03T11:24:16Z)
• Part-guided Relational Transformers for Fine-grained Visual Recognition [59.20531172172135]
  We propose a framework to learn the discriminative part features and explore correlations with a feature transformation module. Our proposed approach does not rely on additional part branches and reaches state-the-of-art performance on 3-of-the-level object recognition.
  arXiv  Detail & Related papers  (2022-12-28T03:45:56Z)
• Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding [95.78002228538841]
  We propose a new open-world semantic segmentation pipeline that makes the first attempt to learn to segment semantic objects of various open-world categories without any efforts on dense annotations. Our method can directly segment objects of arbitrary categories, outperforming zero-shot segmentation methods that require data labeling on three benchmark datasets.
  arXiv  Detail & Related papers  (2022-07-18T09:20:04Z)
• SemAffiNet: Semantic-Affine Transformation for Point Cloud Segmentation [94.11915008006483]
  We propose SemAffiNet for point cloud semantic segmentation. We conduct extensive experiments on the ScanNetV2 and NYUv2 datasets.
  arXiv  Detail & Related papers  (2022-05-26T17:00:23Z)
• A Unified Architecture of Semantic Segmentation and Hierarchical Generative Adversarial Networks for Expression Manipulation [52.911307452212256]
  We develop a unified architecture of semantic segmentation and hierarchical GANs. A unique advantage of our framework is that on forward pass the semantic segmentation network conditions the generative model. We evaluate our method on two challenging facial expression translation benchmarks, AffectNet and RaFD, and a semantic segmentation benchmark, CelebAMask-HQ.
  arXiv  Detail & Related papers  (2021-12-08T22:06:31Z)
• Robust 3D Scene Segmentation through Hierarchical and Learnable Part-Fusion [9.275156524109438]
  3D semantic segmentation is a fundamental building block for several scene understanding applications such as autonomous driving, robotics and AR/VR. Previous methods have utilized hierarchical, iterative methods to fuse semantic and instance information, but they lack learnability in context fusion. This paper presents Segment-Fusion, a novel attention-based method for hierarchical fusion of semantic and instance information.
  arXiv  Detail & Related papers  (2021-11-16T13:14:47Z)
• Affinity Space Adaptation for Semantic Segmentation Across Domains [57.31113934195595]
  In this paper, we address the problem of unsupervised domain adaptation (UDA) in semantic segmentation. Motivated by the fact that source and target domain have invariant semantic structures, we propose to exploit such invariance across domains. We develop two affinity space adaptation strategies: affinity space cleaning and adversarial affinity space alignment.
  arXiv  Detail & Related papers  (2020-09-26T10:28:11Z)
• Improving Semantic Segmentation via Decoupled Body and Edge Supervision [89.57847958016981]
  Existing semantic segmentation approaches either aim to improve the object's inner consistency by modeling the global context, or refine objects detail along their boundaries by multi-scale feature fusion. In this paper, a new paradigm for semantic segmentation is proposed. Our insight is that appealing performance of semantic segmentation requires textitexplicitly modeling the object textitbody and textitedge, which correspond to the high and low frequency of the image. We show that the proposed framework with various baselines or backbone networks leads to better object inner consistency and object boundaries.
  arXiv  Detail & Related papers  (2020-07-20T12:11:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.