ContextFormer: Redefining Efficiency in Semantic Segmentation

• URL: http://arxiv.org/abs/2501.19255v1
• Date: Fri, 31 Jan 2025 16:11:04 GMT
• Title: ContextFormer: Redefining Efficiency in Semantic Segmentation
• Authors: Mian Muhammad Naeem Abid, Nancy Mehta, Zongwei Wu, Fayaz Ali Dharejo, Radu Timofte,
• Abstract summary: Convolutional methods, although capturing local dependencies well, struggle with long-range relationships. Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands. We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation.
• Score: 46.06496660333768
• License:
• Abstract: Semantic segmentation assigns labels to pixels in images, a critical yet challenging task in computer vision. Convolutional methods, although capturing local dependencies well, struggle with long-range relationships. Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands, especially for high-resolution inputs. Most research optimizes the encoder architecture, leaving the bottleneck underexplored - a key area for enhancing performance and efficiency. We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation. The framework's efficiency is driven by three synergistic modules: the Token Pyramid Extraction Module (TPEM) for hierarchical multi-scale representation, the Transformer and Modulating DepthwiseConv (Trans-MDC) block for dynamic scale-aware feature modeling, and the Feature Merging Module (FMM) for robust integration with enhanced spatial and contextual consistency. Extensive experiments on ADE20K, Pascal Context, CityScapes, and COCO-Stuff datasets show ContextFormer significantly outperforms existing models, achieving state-of-the-art mIoU scores, setting a new benchmark for efficiency and performance. The codes will be made publicly available.

Related papers

• Optimized Unet with Attention Mechanism for Multi-Scale Semantic Segmentation [8.443350618722564]
  This paper proposes an improved Unet model combined with an attention mechanism. It introduces channel attention and spatial attention modules, enhances the model's ability to focus on important features. The improved model performs well in terms of mIoU and pixel accuracy (PA), reaching 76.5% and 95.3% respectively.
  arXiv  Detail & Related papers  (2025-02-06T06:51:23Z)
• EMMA: Efficient Visual Alignment in Multi-Modal LLMs [56.03417732498859]
  EMMA is a lightweight cross-modality module designed to efficiently fuse visual and textual encodings. EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations.
  arXiv  Detail & Related papers  (2024-10-02T23:00:31Z)
• CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications [73.80247057590519]
  Vision Transformers (ViTs) mark a revolutionary advance in neural networks with their token mixer's powerful global context capability. We introduce CAS-ViT: Convolutional Additive Self-attention Vision Transformers, to achieve a balance between efficiency and performance in mobile applications. Our model achieves 83.0%/84.1% top-1 with only 12M/21M parameters on ImageNet-1K.
  arXiv  Detail & Related papers  (2024-08-07T11:33:46Z)
• PEM: Prototype-based Efficient MaskFormer for Image Segmentation [10.795762739721294]
  Recent transformer-based architectures have shown impressive results in the field of image segmentation. We propose Prototype-based Efficient MaskFormer (PEM), an efficient transformer-based architecture that can operate in multiple segmentation tasks.
  arXiv  Detail & Related papers  (2024-02-29T18:21:54Z)
• Low-Resolution Self-Attention for Semantic Segmentation [93.30597515880079]
  We introduce the Low-Resolution Self-Attention (LRSA) mechanism to capture global context at a significantly reduced computational cost. Our approach involves computing self-attention in a fixed low-resolution space regardless of the input image's resolution. We demonstrate the effectiveness of our LRSA approach by building the LRFormer, a vision transformer with an encoder-decoder structure.
  arXiv  Detail & Related papers  (2023-10-08T06:10:09Z)
• RTFormer: Efficient Design for Real-Time Semantic Segmentation with Transformer [63.25665813125223]
  We propose RTFormer, an efficient dual-resolution transformer for real-time semantic segmenation. It achieves better trade-off between performance and efficiency than CNN-based models. Experiments on mainstream benchmarks demonstrate the effectiveness of our proposed RTFormer.
  arXiv  Detail & Related papers  (2022-10-13T16:03:53Z)
• ASAP: Accurate semantic segmentation for real time performance [3.5327983932835165]
  We propose an efficient feature fusion method, Feature Fusion with Different Norms (FFDN) FFDN utilizes rich global context of multi-level scale and vertical pooling module before self-attention. We achieve the mean Interaction of-union(mIoU) of 73.1 and the Frame Per Second(FPS) of 191, which are comparable results with state-of-the-arts on Cityscapes test datasets.
  arXiv  Detail & Related papers  (2022-10-04T02:35:53Z)
• Transformer-based Context Condensation for Boosting Feature Pyramids in Object Detection [77.50110439560152]
  Current object detectors typically have a feature pyramid (FP) module for multi-level feature fusion (MFF) We propose a novel and efficient context modeling mechanism that can help existing FPs deliver better MFF results. In particular, we introduce a novel insight that comprehensive contexts can be decomposed and condensed into two types of representations for higher efficiency.
  arXiv  Detail & Related papers  (2022-07-14T01:45:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.