Stochastic Subsampling With Average Pooling

• URL: http://arxiv.org/abs/2409.16630v1
• Date: Wed, 25 Sep 2024 05:18:17 GMT
• Title: Stochastic Subsampling With Average Pooling
• Abstract summary: We propose a new module called average pooling, which incorporates Dropout-likeity in pooling. We describe the properties of subsampling and average pooling and leverage them to design a module without any inconsistency problem. The average pooling achieves a regularization effect without any potential performance degradation due to the inconsistency issue and can easily be plugged into existing deep neural networks.
• Score: 7.229820415732795
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Regularization of deep neural networks has been an important issue to achieve higher generalization performance without overfitting problems. Although the popular method of Dropout provides a regularization effect, it causes inconsistent properties in the output, which may degrade the performance of deep neural networks. In this study, we propose a new module called stochastic average pooling, which incorporates Dropout-like stochasticity in pooling. We describe the properties of stochastic subsampling and average pooling and leverage them to design a module without any inconsistency problem. The stochastic average pooling achieves a regularization effect without any potential performance degradation due to the inconsistency issue and can easily be plugged into existing architectures of deep neural networks. Experiments demonstrate that replacing existing average pooling with stochastic average pooling yields consistent improvements across a variety of tasks, datasets, and models.

Related papers

• FlexPooling with Simple Auxiliary Classifiers in Deep Networks [12.728008452578427]
  Pooling is an essential operation in convolutional neural networks (CNNs)<n>We propose FlexPooling, which generalizes average pooling by learning a weighted average over activations jointly with the rest of the network.<n>Experiments show that FlexPooling consistently outperforms baseline networks, achieving approximately 1 to 3 percent improvement in accuracy.
  arXiv  Detail & Related papers  (2026-06-12T20:04:00Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• From Overfitting to Reliability: Introducing the Hierarchical Approximate Bayesian Neural Network [3.632251954989679]
  HABNN is a novel approach that uses a Gaussian-inverse-Wishart distribution as a hyperprior of the network's weights.<n>Results indicate that HABNN not only matches but often outperforms state-of-the-art models.
  arXiv  Detail & Related papers  (2025-12-15T09:08:42Z)
• Towards noise contrastive estimation with soft targets for conditional models [0.6138671548064356]
  We propose a loss function that is compatible with probabilistic targets. Our new soft target InfoNCE loss is conceptually simple, efficient to compute, and can be motivated through the framework of noise contrastive estimation. We observe that soft target InfoNCE performs on par with strong soft target cross-entropy baselines and outperforms hard target NLL and InfoNCE losses on popular benchmarks.
  arXiv  Detail & Related papers  (2024-04-22T10:45:59Z)
• CoRMF: Criticality-Ordered Recurrent Mean Field Ising Solver [4.364088891019632]
  We propose an RNN-based efficient Ising model solver, the Criticality-ordered Recurrent Mean Field (CoRMF) By leveraging the approximated tree structure of the underlying Ising graph, the newly-obtained criticality order enables the unification between variational mean-field and RNN. CoRFM solves the Ising problems in a self-train fashion without data/evidence, and the inference tasks can be executed by directly sampling from RNN.
  arXiv  Detail & Related papers  (2024-03-05T16:55:06Z)
• Vanishing Feature: Diagnosing Model Merging and Beyond [1.1510009152620668]
  We identify the vanishing feature'' phenomenon, where input-induced features diminish during propagation through a merged model. We show that existing normalization strategies can be enhanced by precisely targeting the vanishing feature issue. We propose the Preserve-First Merging'' (PFM) strategy, which focuses on preserving early-layer features.
  arXiv  Detail & Related papers  (2024-02-05T17:06:26Z)
• Uncertainty-Aware Deep Attention Recurrent Neural Network for Heterogeneous Time Series Imputation [0.25112747242081457]
  Missingness is ubiquitous in multivariate time series and poses an obstacle to reliable downstream analysis. We propose DEep Attention Recurrent Imputation (Imputation), which jointly estimates missing values and their associated uncertainty. Experiments show that I surpasses the SOTA in diverse imputation tasks using real-world datasets.
  arXiv  Detail & Related papers  (2024-01-04T13:21:11Z)
• Model Merging by Uncertainty-Based Gradient Matching [70.54580972266096]
  We propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. Our new method gives consistent improvements for large language models and vision transformers.
  arXiv  Detail & Related papers  (2023-10-19T15:02:45Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Learning Robust Kernel Ensembles with Kernel Average Pooling [3.6540368812166872]
  We introduce Kernel Average Pooling (KAP), a neural network building block that applies the mean filter along the kernel dimension of the layer activation tensor. We show that ensembles of kernels with similar functionality naturally emerge in convolutional neural networks equipped with KAP and trained with backpropagation.
  arXiv  Detail & Related papers  (2022-09-30T19:49:14Z)
• Imbalanced Nodes Classification for Graph Neural Networks Based on Valuable Sample Mining [9.156427521259195]
  A new loss function FD-Loss is reconstructed based on the traditional algorithm-level approach to the imbalance problem. Our loss function can effectively solve the sample node imbalance problem and improve the classification accuracy by 4% compared to existing methods in the node classification task.
  arXiv  Detail & Related papers  (2022-09-18T09:22:32Z)
• ClusterQ: Semantic Feature Distribution Alignment for Data-Free Quantization [111.12063632743013]
  We propose a new and effective data-free quantization method termed ClusterQ. To obtain high inter-class separability of semantic features, we cluster and align the feature distribution statistics. We also incorporate the intra-class variance to solve class-wise mode collapse.
  arXiv  Detail & Related papers  (2022-04-30T06:58:56Z)
• Distribution Mismatch Correction for Improved Robustness in Deep Neural Networks [86.42889611784855]
  normalization methods increase the vulnerability with respect to noise and input corruptions. We propose an unsupervised non-parametric distribution correction method that adapts the activation distribution of each layer. In our experiments, we empirically show that the proposed method effectively reduces the impact of intense image corruptions.
  arXiv  Detail & Related papers  (2021-10-05T11:36:25Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)
• Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks [107.77595511218429]
  In this paper, we investigate the empirical Rademacher complexity related to intermediate layers of deep neural networks. We propose a feature distortion method (Disout) for addressing the aforementioned problem. The superiority of the proposed feature map distortion for producing deep neural network with higher testing performance is analyzed and demonstrated.
  arXiv  Detail & Related papers  (2020-02-23T13:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.