SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation

• URL: http://arxiv.org/abs/2011.12025v2
• Date: Fri, 5 Aug 2022 15:46:02 GMT
• Title: SegBlocks: Block-Based Dynamic Resolution Networks for Real-Time Segmentation
• Authors: Thomas Verelst and Tinne Tuytelaars
• Abstract summary: SegBlocks dynamically adjusts the processing resolution of image regions based on their complexity. A lightweight policy network, selecting the complex regions, is trained using reinforcement learning. Our method reduces the number of floating-point operations of SwiftNet-RN18 by 60% and increases the inference speed by 50%.
• Score: 47.338987325018614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: SegBlocks reduces the computational cost of existing neural networks, by dynamically adjusting the processing resolution of image regions based on their complexity. Our method splits an image into blocks and downsamples blocks of low complexity, reducing the number of operations and memory consumption. A lightweight policy network, selecting the complex regions, is trained using reinforcement learning. In addition, we introduce several modules implemented in CUDA to process images in blocks. Most important, our novel BlockPad module prevents the feature discontinuities at block borders of which existing methods suffer, while keeping memory consumption under control. Our experiments on Cityscapes, Camvid and Mapillary Vistas datasets for semantic segmentation show that dynamically processing images offers a better accuracy versus complexity trade-off compared to static baselines of similar complexity. For instance, our method reduces the number of floating-point operations of SwiftNet-RN18 by 60% and increases the inference speed by 50%, with only 0.3% decrease in mIoU accuracy on Cityscapes.

Related papers

• Moiré Zero: An Efficient and High-Performance Neural Architecture for Moiré Removal [8.464291713830127]
  We propose MZNet, a U-shaped network designed to bring images closer to a 'Moire-Zero' state by effectively removing moir'e patterns.<n>MZNet achieves state-of-the-art performance on high-resolution datasets and delivers competitive results on lower-resolution dataset.
  arXiv  Detail & Related papers  (2025-07-30T06:16:35Z)
• High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution [87.56382172827526]
  High-frequency regions are most critical for reconstruction.<n>We propose a training-free adaptive masking module for acceleration.<n>Our method reduces FLOPs by 24--43% for state-of-the-art models.
  arXiv  Detail & Related papers  (2025-05-11T13:18:03Z)
• CATANet: Efficient Content-Aware Token Aggregation for Lightweight Image Super-Resolution [42.76046559103463]
  Transformer-based methods have demonstrated impressive performance in low-level visual tasks such as Image Super-Resolution (SR) These methods limit attention to content-agnostic local regions, limiting directly the ability of attention to capture long-range dependency. We propose a lightweight Content-Aware Token Aggregation Network (CATANet) to address these issues. Our method achieves superior performance, with a maximum PSNR improvement of 0.33dB and nearly double the inference speed.
  arXiv  Detail & Related papers  (2025-03-10T04:00:27Z)
• CubeFormer: A Simple yet Effective Baseline for Lightweight Image Super-Resolution [55.94314421887744]
  Lightweight image super-resolution (SR) methods aim at increasing the resolution and restoring the details of an image using a lightweight neural network. Our analysis reveals that these methods are hindered by constrained feature diversity, which adversely impacts feature representation and detail recovery. We propose a simple yet effective baseline called CubeFormer, designed to enhance feature richness by completing holistic information aggregation.
  arXiv  Detail & Related papers  (2024-12-03T08:02:26Z)
• UTSRMorph: A Unified Transformer and Superresolution Network for Unsupervised Medical Image Registration [4.068692674719378]
  Complicated image registration is a key issue in medical image analysis. We propose a novel unsupervised image registration method named the unified Transformer and superresolution (UTSRMorph) network.
  arXiv  Detail & Related papers  (2024-10-27T06:28:43Z)
• Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss [59.835032408496545]
  We propose a tile-based strategy that partitions the contrastive loss calculation into arbitrary small blocks. We also introduce a multi-level tiling strategy to leverage the hierarchical structure of distributed systems. Compared to SOTA memory-efficient solutions, it achieves a two-order-of-magnitude reduction in memory while maintaining comparable speed.
  arXiv  Detail & Related papers  (2024-10-22T17:59:30Z)
• Serpent: Scalable and Efficient Image Restoration via Multi-scale Structured State Space Models [22.702352459581434]
  Serpent is an efficient architecture for high-resolution image restoration. We show that Serpent can achieve reconstruction quality on par with state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-03-26T17:43:15Z)
• Cache Me if You Can: Accelerating Diffusion Models through Block Caching [67.54820800003375]
  A large image-to-image network has to be applied many times to iteratively refine an image from random noise. We investigate the behavior of the layers within the network and find that 1) the layers' output changes smoothly over time, 2) the layers show distinct patterns of change, and 3) the change from step to step is often very small. We propose a technique to automatically determine caching schedules based on each block's changes over timesteps.
  arXiv  Detail & Related papers  (2023-12-06T00:51:38Z)
• Pixel Adapter: A Graph-Based Post-Processing Approach for Scene Text Image Super-Resolution [22.60056946339325]
  We propose the Pixel Adapter Module (PAM) based on graph attention to address pixel distortion caused by upsampling. The PAM effectively captures local structural information by allowing each pixel to interact with its neighbors and update features. We demonstrate that our proposed method generates high-quality super-resolution images, surpassing existing methods in recognition accuracy.
  arXiv  Detail & Related papers  (2023-09-16T08:12:12Z)
• Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution [90.16462805389943]
  We develop a spatially-adaptive feature modulation (SAFM) mechanism upon a vision transformer (ViT)-like block. Proposed method is $3times$ smaller than state-of-the-art efficient SR methods.
  arXiv  Detail & Related papers  (2023-02-27T14:19:31Z)
• Accurate Image Restoration with Attention Retractable Transformer [50.05204240159985]
  We propose Attention Retractable Transformer (ART) for image restoration. ART presents both dense and sparse attention modules in the network. We conduct extensive experiments on image super-resolution, denoising, and JPEG compression artifact reduction tasks.
  arXiv  Detail & Related papers  (2022-10-04T07:35:01Z)
• BlockCopy: High-Resolution Video Processing with Block-Sparse Feature Propagation and Online Policies [57.62315799929681]
  BlockCopy is a scheme that accelerates pretrained frame-based CNNs to process video more efficiently. A lightweight policy network determines important regions in an image, and operations are applied on selected regions only. Features of non-selected regions are simply copied from the preceding frame, reducing the number of computations and latency.
  arXiv  Detail & Related papers  (2021-08-20T21:16:01Z)
• PNEN: Pyramid Non-Local Enhanced Networks [23.17149002568982]
  We propose a novel non-local module, Pyramid Non-local Block, to build up connection between every pixel and all remain pixels. Based on the proposed module, we devise a Pyramid Non-local Enhanced Networks for edge-preserving image smoothing. We integrate it into two existing methods for image denoising and single image super-resolution, achieving consistently improved performance.
  arXiv  Detail & Related papers  (2020-08-22T03:10:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.