Balanced Group Convolution: An Improved Group Convolution Based on Approximability Estimates

• URL: http://arxiv.org/abs/2310.12461v1
• Date: Thu, 19 Oct 2023 04:39:38 GMT
• Title: Balanced Group Convolution: An Improved Group Convolution Based on Approximability Estimates
• Authors: Youngkyu Lee, Jongho Park, Chang-Ock Lee
• Abstract summary: Group convolution effectively reduces the computational cost by grouping channels. We mathematically analyze the approximation of the group convolution to the standard convolution. We propose a novel variant of the group convolution called balanced group convolution, which shows a higher approximation with a small additional computational cost.
• Score: 1.927926533063962
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The performance of neural networks has been significantly improved by increasing the number of channels in convolutional layers. However, this increase in performance comes with a higher computational cost, resulting in numerous studies focused on reducing it. One promising approach to address this issue is group convolution, which effectively reduces the computational cost by grouping channels. However, to the best of our knowledge, there has been no theoretical analysis on how well the group convolution approximates the standard convolution. In this paper, we mathematically analyze the approximation of the group convolution to the standard convolution with respect to the number of groups. Furthermore, we propose a novel variant of the group convolution called balanced group convolution, which shows a higher approximation with a small additional computational cost. We provide experimental results that validate our theoretical findings and demonstrate the superior performance of the balanced group convolution over other variants of group convolution.

Related papers

• GroupedMixer: An Entropy Model with Group-wise Token-Mixers for Learned Image Compression [64.47244912937204]
  We propose a novel transformer-based entropy model called GroupedMixer. GroupedMixer enjoys both faster coding speed and better compression performance than previous transformer-based methods. Experimental results demonstrate that the proposed GroupedMixer yields the state-of-the-art rate-distortion performance with fast compression speed.
  arXiv  Detail & Related papers  (2024-05-02T10:48:22Z)
• How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance [64.1656365676171]
  Group imbalance has been a known problem in empirical risk minimization. This paper quantifies the impact of individual groups on the sample complexity, the convergence rate, and the average and group-level testing performance.
  arXiv  Detail & Related papers  (2024-03-12T04:38:05Z)
• Achieving Sample and Computational Efficient Reinforcement Learning by Action Space Reduction via Grouping [7.691755449724638]
  Reinforcement learning often needs to deal with the exponential growth of states and actions in high-dimensional spaces. We learn the inherent structure of action-wise similar MDP to appropriately balance the performance degradation versus sample/computational complexity.
  arXiv  Detail & Related papers  (2023-06-22T15:40:10Z)
• Sparse-group boosting -- Unbiased group and variable selection [0.0]
  We show that within-group and between-group sparsity can be controlled by a mixing parameter. With simulations, gene data as well as agricultural data we show the effectiveness and predictive competitiveness of this estimator.
  arXiv  Detail & Related papers  (2022-06-13T17:44:16Z)
• Exploiting Redundancy: Separable Group Convolutional Networks on Lie Groups [14.029933823101084]
  Group convolutional neural networks (G-CNNs) have been shown to increase parameter efficiency and model accuracy. In this work, we investigate the properties of representations learned by regular G-CNNs, and show considerable parameter redundancy in group convolution kernels. We introduce convolution kernels that are separable over the subgroup and channel dimensions.
  arXiv  Detail & Related papers  (2021-10-25T15:56:53Z)
• Two-level Group Convolution [2.2344764434954256]
  Group convolution has been widely used in order to reduce the computation time of convolution. We propose a new convolution methodology called two-level'' group convolution that is robust with respect to the increase of the number of groups.
  arXiv  Detail & Related papers  (2021-10-11T07:54:49Z)
• Harnessing Heterogeneity: Learning from Decomposed Feedback in Bayesian Modeling [68.69431580852535]
  We introduce a novel GP regression to incorporate the subgroup feedback. Our modified regression has provably lower variance -- and thus a more accurate posterior -- compared to previous approaches. We execute our algorithm on two disparate social problems.
  arXiv  Detail & Related papers  (2021-07-07T03:57:22Z)
• Group Equivariant Neural Architecture Search via Group Decomposition and Reinforcement Learning [17.291131923335918]
  We prove a new group-theoretic result in the context of equivariant neural networks. We also design an algorithm to construct equivariant networks that significantly improves computational complexity. We use deep Q-learning to search for group equivariant networks that maximize performance.
  arXiv  Detail & Related papers  (2021-04-10T19:37:25Z)
• Partition-based formulations for mixed-integer optimization of trained ReLU neural networks [66.88252321870085]
  This paper introduces a class of mixed-integer formulations for trained ReLU neural networks. At one extreme, one partition per input recovers the convex hull of a node, i.e., the tightest possible formulation for each node.
  arXiv  Detail & Related papers  (2021-02-08T17:27:34Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)
• Stochastic Flows and Geometric Optimization on the Orthogonal Group [52.50121190744979]
  We present a new class of geometrically-driven optimization algorithms on the orthogonal group $O(d)$. We show that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, flows and metric learning.
  arXiv  Detail & Related papers  (2020-03-30T15:37:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.