Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation

• URL: http://arxiv.org/abs/2604.13761v1
• Date: Wed, 15 Apr 2026 11:47:34 GMT
• Title: Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation
• Abstract summary: Sparse mixture-of-experts (MoE) layers have been shown to substantially increase model capacity without a proportional increase in computational cost.<n>Integrating sparse MoE layers into convolutional neural networks (CNNs) remains inconsistent.<n>In this work, we investigate a coarser, patch-wise formulation of sparse MoE layers for semantic segmentation.
• Score: 10.912224105652044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse mixture-of-experts (MoE) layers have been shown to substantially increase model capacity without a proportional increase in computational cost and are widely used in transformer architectures, where they typically replace feed-forward network blocks. In contrast, integrating sparse MoE layers into convolutional neural networks (CNNs) remains inconsistent, with most prior work focusing on fine-grained MoEs operating at the filter or channel levels. In this work, we investigate a coarser, patch-wise formulation of sparse MoE layers for semantic segmentation, where local regions are routed to a small subset of convolutional experts. Through experiments on the Cityscapes and BDD100K datasets using encoder-decoder and backbone-based CNNs, we conduct a design analysis to assess how architectural choices affect routing dynamics and expert specialization. Our results demonstrate consistent, architecture-dependent improvements (up to +3.9 mIoU) with little computational overhead, while revealing strong design sensitivity. Our work provides empirical insights into the design and internal dynamics of sparse MoE layers in CNN-based dense prediction. Our code is available at https://github.com/KASTEL-MobilityLab/moe-layers/.

Related papers

• LayerTracer: A Joint Task-Particle and Vulnerable-Layer Analysis framework for Arbitrary Large Language Model Architectures [2.175079806379993]
  This paper proposes LayerTracer, an architecture-agnostic end-to-end analysis framework compatible with any Large Language Models (LLMs) architecture.<n>By extracting hidden states layer-by-layer and mapping them to vocabulary probability distributions, it achieves joint analysis of task particle localization and layer vulnerability quantification.
  arXiv  Detail & Related papers  (2026-04-22T13:38:15Z)
• GCRPNet: Graph-Enhanced Contextual and Regional Perception Network for Salient Object Detection in Optical Remote Sensing Images [68.33481681452675]
  We propose a graph-enhanced contextual and regional perception network (GCRPNet)<n>It builds upon the Mamba architecture to simultaneously capture long-range dependencies and enhance regional feature representation.<n>It performs adaptive patch scanning on feature maps processed via multi-scale convolutions, thereby capturing rich local region information.
  arXiv  Detail & Related papers  (2025-08-14T11:31:43Z)
• ASNN: Learning to Suggest Neural Architectures from Performance Distributions [0.0]
  Architecture of a neural network (NN) plays a critical role in determining its performance.<n>There is no general closed-form function that maps between network structure and accuracy.<n>We propose the Architecture Suggesting Network (ASNN), a model designed to learn the relationship between NN architecture and its test accuracy.
  arXiv  Detail & Related papers  (2025-07-27T07:39:33Z)
• AMBER: Adaptive Mesh Generation by Iterative Mesh Resolution Prediction [48.72179728638418]
  We propose Adaptive Meshing By Expert Reconstruction (AMBER), a supervised learning approach to mesh adaptation.<n>AMBER iteratively predicts the sizing field, and uses this prediction to produce a new intermediate mesh using an out-of-the-box mesh generator.<n>We evaluate AMBER on 2D and 3D geometries, datasets including classical physics problems, mechanical components, and real-world industrial designs with human expert meshes.
  arXiv  Detail & Related papers  (2025-05-29T17:10:44Z)
• Mechanistic Design and Scaling of Hybrid Architectures [114.3129802943915]
  We identify and test new hybrid architectures constructed from a variety of computational primitives. We experimentally validate the resulting architectures via an extensive compute-optimal and a new state-optimal scaling law analysis. We find MAD synthetics to correlate with compute-optimal perplexity, enabling accurate evaluation of new architectures.
  arXiv  Detail & Related papers  (2024-03-26T16:33:12Z)
• Dynamic Semantic Compression for CNN Inference in Multi-access Edge Computing: A Graph Reinforcement Learning-based Autoencoder [82.8833476520429]
  We propose a novel semantic compression method, autoencoder-based CNN architecture (AECNN) for effective semantic extraction and compression in partial offloading. In the semantic encoder, we introduce a feature compression module based on the channel attention mechanism in CNNs, to compress intermediate data by selecting the most informative features. In the semantic decoder, we design a lightweight decoder to reconstruct the intermediate data through learning from the received compressed data to improve accuracy.
  arXiv  Detail & Related papers  (2024-01-19T15:19:47Z)
• Adaptive Growth: Real-time CNN Layer Expansion [0.0]
  This research presents a new algorithm that allows the convolutional layer of a Convolutional Neural Network (CNN) to dynamically evolve based on data input. Instead of a rigid architecture, our approach iteratively introduces kernels to the convolutional layer, gauging its real-time response to varying data. Remarkably, our unsupervised method has outstripped its supervised counterparts across diverse datasets.
  arXiv  Detail & Related papers  (2023-09-06T14:43:58Z)
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures [179.66117325866585]
  We investigate a design space that is usually overlooked, i.e. adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by shrinking widened baseline networks and leads to superior performance. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration.
  arXiv  Detail & Related papers  (2020-06-29T17:59:26Z)
• When Residual Learning Meets Dense Aggregation: Rethinking the Aggregation of Deep Neural Networks [57.0502745301132]
  We propose Micro-Dense Nets, a novel architecture with global residual learning and local micro-dense aggregations. Our micro-dense block can be integrated with neural architecture search based models to boost their performance.
  arXiv  Detail & Related papers  (2020-04-19T08:34:52Z)
• Rethinking Depthwise Separable Convolutions: How Intra-Kernel Correlations Lead to Improved MobileNets [6.09170287691728]
  We introduce blueprint separable convolutions (BSConv) as highly efficient building blocks for CNNs. They are motivated by quantitative analyses of kernel properties from trained models. Our approach provides a thorough theoretical derivation, interpretation, and justification for the application of depthwise separable convolutions.
  arXiv  Detail & Related papers  (2020-03-30T15:23:27Z)
• Inferring Convolutional Neural Networks' accuracies from their architectural characterizations [0.0]
  We study the relationships between a CNN's architecture and its performance. We show that the attributes can be predictive of the networks' performance in two specific computer vision-based physics problems. We use machine learning models to predict whether a network can perform better than a certain threshold accuracy before training.
  arXiv  Detail & Related papers  (2020-01-07T16:41:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.