Quotient Network -- A Network Similar to ResNet but Learning Quotients

• URL: http://arxiv.org/abs/2506.00992v1
• Date: Sun, 01 Jun 2025 12:46:43 GMT
• Title: Quotient Network -- A Network Similar to ResNet but Learning Quotients
• Authors: Peng Hui, Jiamuyang Zhao, Changxin Li, Qingzhen Zhu,
• Abstract summary: ResNet is a powerful tool for training extremely deep networks.<n>We propose a new network that perfectly solves these two problems while still having the advantages of ResNet.<n>Specifically, it chooses to learn the quotient of the target features with the existing features, so we call it the quotient network.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The emergence of ResNet provides a powerful tool for training extremely deep networks. The core idea behind it is to change the learning goals of the network. It no longer learns new features from scratch but learns the difference between the target and existing features. However, the difference between the two kinds of features does not have an independent and clear meaning, and the amount of learning is based on the absolute rather than the relative difference, which is sensitive to the size of existing features. We propose a new network that perfectly solves these two problems while still having the advantages of ResNet. Specifically, it chooses to learn the quotient of the target features with the existing features, so we call it the quotient network. In order to enable this network to learn successfully and achieve higher performance, we propose some design rules for this network so that it can be trained efficiently and achieve better performance than ResNet. Experiments on the CIFAR10, CIFAR100, and SVHN datasets prove that this network can stably achieve considerable improvements over ResNet by simply making tiny corresponding changes to the original ResNet network without adding new parameters.

Related papers

• Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers [83.74380713308605]
  We develop a new type of transformation that is fully compatible with a variant of ReLUs -- Leaky ReLUs. We show in experiments that our method, which introduces negligible extra computational cost, validation accuracies with deep vanilla networks that are competitive with ResNets.
  arXiv  Detail & Related papers  (2022-03-15T17:49:08Z)
• Highly Accurate and Reliable Wireless Network Slicing in 5th Generation Networks: A Hybrid Deep Learning Approach [21.137037568638974]
  We propose a hybrid deep learning model that consists of a convolution neural network (CNN) and long short term memory (LSTM) The overall accuracy of 95.17% is achieved by the proposed model that reflects its applicability.
  arXiv  Detail & Related papers  (2021-10-07T10:05:25Z)
• Learning distinct features helps, provably [98.78384185493624]
  We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average $L$-distance between the hidden-layer features.
  arXiv  Detail & Related papers  (2021-06-10T19:14:45Z)
• Why Do Better Loss Functions Lead to Less Transferable Features? [93.47297944685114]
  This paper studies how the choice of training objective affects the transferability of the hidden representations of convolutional neural networks trained on ImageNet. We show that many objectives lead to statistically significant improvements in ImageNet accuracy over vanilla softmax cross-entropy, but the resulting fixed feature extractors transfer substantially worse to downstream tasks.
  arXiv  Detail & Related papers  (2020-10-30T17:50:31Z)
• ResNet or DenseNet? Introducing Dense Shortcuts to ResNet [80.35001540483789]
  This paper presents a unified perspective of dense summation to analyze them. We propose dense weighted normalized shortcuts as a solution to the dilemma between ResNet and DenseNet. Our proposed DSNet achieves significantly better results than ResNet, and achieves comparable performance as DenseNet but requiring fewer resources.
  arXiv  Detail & Related papers  (2020-10-23T16:00:15Z)
• Adjoined Networks: A Training Paradigm with Applications to Network Compression [3.995047443480282]
  We introduce Adjoined Networks, or AN, a learning paradigm that trains both the original base network and the smaller compressed network together. Using ResNet-50 as the base network, AN achieves 71.8% top-1 accuracy with only 1.8M parameters and 1.6 GFLOPs on the ImageNet data-set. We propose Differentiable Adjoined Networks (DAN), a training paradigm that augments AN by using neural architecture search to jointly learn both the width and the weights for each layer of the smaller network.
  arXiv  Detail & Related papers  (2020-06-10T02:48:16Z)
• Improved Residual Networks for Image and Video Recognition [98.10703825716142]
  Residual networks (ResNets) represent a powerful type of convolutional neural network (CNN) architecture. We show consistent improvements in accuracy and learning convergence over the baseline. Our proposed approach allows us to train extremely deep networks, while the baseline shows severe optimization issues.
  arXiv  Detail & Related papers  (2020-04-10T11:09:50Z)
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
  This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-06T15:51:00Z)
• Knapsack Pruning with Inner Distillation [11.04321604965426]
  We propose a novel pruning method that optimize the final accuracy of the pruned network. We prune the network channels while maintaining the high-level structure of the network. Our method leads to state-of-the-art pruning results on ImageNet, CIFAR-10 and CIFAR-100 using ResNet backbones.
  arXiv  Detail & Related papers  (2020-02-19T16:04:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.