Sparse Hybrid Linear-Morphological Networks

• URL: http://arxiv.org/abs/2504.09289v1
• Date: Sat, 12 Apr 2025 17:19:46 GMT
• Title: Sparse Hybrid Linear-Morphological Networks
• Authors: Konstantinos Fotopoulos, Christos Garoufis, Petros Maragos,
• Abstract summary: We propose a hybrid network structure, wherein morphological layers are inserted between the linear layers of the network.<n>We conduct experiments on the Magna-Tag-A-Tune (music auto-tagging) and CIFAR-10 (image classification) datasets.<n>We demonstrate that these networks induce sparsity to their linear layers, making them more prunable under L1 unstructured pruning.
• Score: 22.57224128086205
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate hybrid linear-morphological networks. Recent studies highlight the inherent affinity of morphological layers to pruning, but also their difficulty in training. We propose a hybrid network structure, wherein morphological layers are inserted between the linear layers of the network, in place of activation functions. We experiment with the following morphological layers: 1) maxout pooling layers (as a special case of a morphological layer), 2) fully connected dense morphological layers, and 3) a novel, sparsely initialized variant of (2). We conduct experiments on the Magna-Tag-A-Tune (music auto-tagging) and CIFAR-10 (image classification) datasets, replacing the linear classification heads of state-of-the-art convolutional network architectures with our proposed network structure for the various morphological layers. We demonstrate that these networks induce sparsity to their linear layers, making them more prunable under L1 unstructured pruning. We also show that on MTAT our proposed sparsely initialized layer achieves slightly better performance than ReLU, maxout, and densely initialized max-plus layers, and exhibits faster initial convergence.

Related papers

• Training Deep Morphological Neural Networks as Universal Approximators [23.122183338862687]
  We investigate deep morphological neural networks (DMNNs)<n>We demonstrate that activations between layers are essential for DMNNs.<n>We propose several new architectures for DMNNs, each with a different constraint on their parameters.
  arXiv  Detail & Related papers  (2025-05-14T18:10:49Z)
• SIMAP: A simplicial-map layer for neural networks [0.196629787330046]
  The SIMAP layer is an enhanced version of Simplicial-Map Neural Networks (SMNNs) Unlike SMNNs, the support set is based on a fixed maximal simplex, the barycentric subdivision being efficiently computed with a matrix-based multiplication algorithm.
  arXiv  Detail & Related papers  (2024-03-22T10:06:42Z)
• Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination [33.273226655730326]
  We show that each layer of a deep linear network progressively compresses within-class features at a geometric rate and discriminates between-class features at a linear rate. This is the first quantitative characterization of feature evolution in hierarchical representations of deep linear networks.
  arXiv  Detail & Related papers  (2023-11-06T09:00:38Z)
• HaarNet: Large-scale Linear-Morphological Hybrid Network for RGB-D Semantic Segmentation [12.89384111017003]
  This is the first large-scale linear-morphological hybrid evaluated on a set of sizeable real-world datasets. HaarNet is competitive with a state-of-the-art CNN, implying that morphological networks are a promising research direction for geometry-based learning tasks.
  arXiv  Detail & Related papers  (2023-10-11T17:18:15Z)
• Layer-wise Linear Mode Connectivity [52.6945036534469]
  Averaging neural network parameters is an intuitive method for the knowledge of two independent models. It is most prominently used in federated learning. We analyse the performance of the models that result from averaging single, or groups.
  arXiv  Detail & Related papers  (2023-07-13T09:39:10Z)
• WLD-Reg: A Data-dependent Within-layer Diversity Regularizer [98.78384185493624]
  Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization. We propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks.
  arXiv  Detail & Related papers  (2023-01-03T20:57:22Z)
• Improved Convergence Guarantees for Shallow Neural Networks [91.3755431537592]
  We prove convergence of depth 2 neural networks, trained via gradient descent, to a global minimum. Our model has the following features: regression with quadratic loss function, fully connected feedforward architecture, RelU activations, Gaussian data instances, adversarial labels. They strongly suggest that, at least in our model, the convergence phenomenon extends well beyond the NTK regime''
  arXiv  Detail & Related papers  (2022-12-05T14:47:52Z)
• Advances in the training, pruning and enforcement of shape constraints of Morphological Neural Networks using Tropical Algebra [40.327435646554115]
  We study neural networks based on the morphological operators of dilation and erosion. Our contributions include the training of morphological networks via Difference-of-Convex programming methods and extend a binary morphological to multiclass tasks.
  arXiv  Detail & Related papers  (2020-11-15T22:44:25Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures [179.66117325866585]
  We investigate a design space that is usually overlooked, i.e. adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by shrinking widened baseline networks and leads to superior performance. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration.
  arXiv  Detail & Related papers  (2020-06-29T17:59:26Z)
• Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation [2.654526698055524]
  We build an equivalent network fusing normalization layers and convolutional or fully connected layers. Heatmaps obtained with our method on MNIST and CIFAR 10 datasets are more accurate for convolutional layers.
  arXiv  Detail & Related papers  (2020-02-24T13:06:55Z)
• Convolutional Networks with Dense Connectivity [59.30634544498946]
  We introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. We evaluate our proposed architecture on four highly competitive object recognition benchmark tasks.
  arXiv  Detail & Related papers  (2020-01-08T06:54:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.