The Theoretical Expressiveness of Maxpooling

• URL: http://arxiv.org/abs/2203.01016v1
• Date: Wed, 2 Mar 2022 10:45:53 GMT
• Title: The Theoretical Expressiveness of Maxpooling
• Authors: Kyle Matoba and Nikolaos Dimitriadis and Fran\c{c}ois Fleuret
• Abstract summary: We develop a theoretical framework analyzing ReLU based approximations to max pooling. We find that max pooling cannot be efficiently replicated using ReLU activations. We conclude that the main cause of a difference between max pooling and an optimal approximation, can be overcome with other architectural decisions.
• Score: 4.028503203417233
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Over the decade since deep neural networks became state of the art image classifiers there has been a tendency towards less use of max pooling: the function that takes the largest of nearby pixels in an image. Since max pooling featured prominently in earlier generations of image classifiers, we wish to understand this trend, and whether it is justified. We develop a theoretical framework analyzing ReLU based approximations to max pooling, and prove a sense in which max pooling cannot be efficiently replicated using ReLU activations. We analyze the error of a class of optimal approximations, and find that whilst the error can be made exponentially small in the kernel size, doing so requires an exponentially complex approximation. Our work gives a theoretical basis for understanding the trend away from max pooling in newer architectures. We conclude that the main cause of a difference between max pooling and an optimal approximation, a prevalent large difference between the max and other values within pools, can be overcome with other architectural decisions, or is not prevalent in natural images.

Related papers

• Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think [53.2706196341054]
  We show that the perceived inefficiency was caused by a flaw in the inference pipeline that has so far gone unnoticed. We perform end-to-end fine-tuning on top of the single-step model with task-specific losses and get a deterministic model that outperforms all other diffusion-based depth and normal estimation models.
  arXiv  Detail & Related papers  (2024-09-17T16:58:52Z)
• Enhancing Classifier Conservativeness and Robustness by Polynomiality [23.099278014212146]
  We show howconditionality can remedy the situation. A directly related, simple, yet important technical novelty we subsequently present is softRmax. We show that two aspects of softRmax, conservativeness and inherent robustness, lead to adversarial regularization.
  arXiv  Detail & Related papers  (2022-03-23T19:36:19Z)
• AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling [82.08631594071656]
  Pooling layers are essential building blocks of Convolutional Neural Networks (CNNs) We propose an adaptive and exponentially weighted pooling method named adaPool. We demonstrate how adaPool improves the preservation of detail through a range of tasks including image and video classification and object detection.
  arXiv  Detail & Related papers  (2021-11-01T08:50:37Z)
• PixelPyramids: Exact Inference Models from Lossless Image Pyramids [58.949070311990916]
  Pixel-Pyramids is a block-autoregressive approach with scale-specific representations to encode the joint distribution of image pixels. It yields state-of-the-art results for density estimation on various image datasets, especially for high-resolution data. For CelebA-HQ 1024 x 1024, we observe that the density estimates are improved to 44% of the baseline despite sampling speeds superior even to easily parallelizable flow-based models.
  arXiv  Detail & Related papers  (2021-10-17T10:47:29Z)
• Minimax Optimization with Smooth Algorithmic Adversaries [59.47122537182611]
  We propose a new algorithm for the min-player against smooth algorithms deployed by an adversary. Our algorithm is guaranteed to make monotonic progress having no limit cycles, and to find an appropriate number of gradient ascents.
  arXiv  Detail & Related papers  (2021-06-02T22:03:36Z)
• InfinityGAN: Towards Infinite-Resolution Image Synthesis [92.40782797030977]
  We present InfinityGAN, a method to generate arbitrary-resolution images. We show how it trains and infers patch-by-patch seamlessly with low computational resources.
  arXiv  Detail & Related papers  (2021-04-08T17:59:30Z)
• Consensus Maximisation Using Influences of Monotone Boolean Functions [40.86597150734384]
  MaxCon aims to find the largest subset of data that fits the model within some tolerance level. We show that influences of points belonging to the largest structure in data would generally be smaller under certain conditions. Results for both synthetic and real visual data experiments show that the MBF based algorithm is capable of generating a near optimal solution relatively quickly.
  arXiv  Detail & Related papers  (2021-03-06T22:01:06Z)
• Comparison of Methods Generalizing Max- and Average-Pooling [1.693200946453174]
  Max- and average-pooling are the most popular methods for downsampling in convolutional neural networks. In this paper, we compare different pooling methods that generalize both max- and average-pooling. The results show that none of the more sophisticated methods perform significantly better in this classification task than standard max- or average-pooling.
  arXiv  Detail & Related papers  (2021-03-02T14:26:51Z)
• Maximal function pooling with applications [4.446564162927513]
  Maxfun pooling is inspired by the Hardy-Littlewood maximal function. It is presented as a viable alternative to some of the most popular pooling functions, such as max pooling and average pooling. We demonstrate the features of maxfun pooling with two applications: first in the context of convolutional sparse coding, and then for image classification.
  arXiv  Detail & Related papers  (2021-03-01T20:30:04Z)
• Taming GANs with Lookahead-Minmax [63.90038365274479]
  Experimental results on MNIST, SVHN, CIFAR-10, and ImageNet demonstrate a clear advantage of combining Lookahead-minmax with Adam or extragradient. Using 30-fold fewer parameters and 16-fold smaller minibatches we outperform the reported performance of the class-dependent BigGAN on CIFAR-10 by obtaining FID of 12.19 without using the class labels.
  arXiv  Detail & Related papers  (2020-06-25T17:13:23Z)
• An Optimization and Generalization Analysis for Max-Pooling Networks [34.58092926599547]
  Max-Pooling operations are a core component of deep learning architectures. We perform a theoretical analysis of a convolutional max-pooling architecture. We empirically validate that CNNs significantly outperform fully connected networks in our setting.
  arXiv  Detail & Related papers  (2020-02-22T22:26:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.