PENEX: AdaBoost-Inspired Neural Network Regularization

• URL: http://arxiv.org/abs/2510.02107v2
• Date: Mon, 06 Oct 2025 17:51:59 GMT
• Title: PENEX: AdaBoost-Inspired Neural Network Regularization
• Authors: Klaus-Rudolf Kladny, Bernhard Schölkopf, Michael Muehlebach,
• Abstract summary: We introduce PENEX, a new formulation of the multi-class exponential loss.<n>We show that PENEX implicitly maximizes margins of data points.<n>Our results highlight PENEX's potential as an AdaBoost-inspired alternative for effective training and fine-tuning of deep neural networks.
• Score: 53.628953848091065
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AdaBoost sequentially fits so-called weak learners to minimize an exponential loss, which penalizes mislabeled data points more severely than other loss functions like cross-entropy. Paradoxically, AdaBoost generalizes well in practice as the number of weak learners grows. In the present work, we introduce Penalized Exponential Loss (PENEX), a new formulation of the multi-class exponential loss that is theoretically grounded and, in contrast to the existing formulation, amenable to optimization via first-order methods. We demonstrate both empirically and theoretically that PENEX implicitly maximizes margins of data points. Also, we show that gradient increments on PENEX implicitly parameterize weak learners in the boosting framework. Across computer vision and language tasks, we show that PENEX exhibits a regularizing effect often better than established methods with similar computational cost. Our results highlight PENEX's potential as an AdaBoost-inspired alternative for effective training and fine-tuning of deep neural networks.

Related papers

• Graph Unlearning Meets Influence-aware Negative Preference Optimization [45.33243345077153]
  In this paper, we introduce textbfINPO, an textbfInfluence-aware textbfNegative textbfPreference textbfOptimization framework.<n>We first analyze that NPO has slower divergence speed and theoretically propose that unlearning high-influence edges can reduce impact of unlearning.<n>Experiments conducted on five real-world datasets demonstrate that INPO-based model achieves state-of-the-art performance on all forget quality metrics.
  arXiv  Detail & Related papers  (2025-10-22T11:18:00Z)
• Instance-Prototype Affinity Learning for Non-Exemplar Continual Graph Learning [7.821213342456415]
  Graph Neural Networks endure catastrophic forgetting, undermining their capacity to preserve previously acquired knowledge.<n>We propose Instance-Prototype Affinity Learning (IPAL), a novel paradigm for Non-Exemplar Continual Graph Learning (NECGL)<n>We embed a Decision Boundary Perception mechanism within PCL, fostering greater inter-class discriminability.
  arXiv  Detail & Related papers  (2025-05-15T07:35:27Z)
• Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models [52.8949080772873]
  We propose an evolution-based region adversarial prompt tuning method called ER-APT.<n>In each training iteration, we first generate AEs using traditional gradient-based methods.<n> Subsequently, a genetic evolution mechanism incorporating selection, mutation, and crossover is applied to optimize the AEs.<n>The final evolved AEs are used for prompt tuning, achieving region-based adversarial optimization instead of conventional single-point adversarial prompt tuning.
  arXiv  Detail & Related papers  (2025-03-17T07:08:47Z)
• Multiplicative Learning [0.04499833362998487]
  We introduce Expectation Reflection (ER), a novel learning approach that updates weights multiplicatively based on the ratio of observed to predicted outputs.<n>We extend ER to multilayer networks and demonstrate its effectiveness in performing image classification tasks.
  arXiv  Detail & Related papers  (2025-03-13T08:14:00Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Deep Augmentation: Dropout as Augmentation for Self-Supervised Learning [19.495587566796278]
  Deep Augmentation is a method that applies dropout or PCA transformations to targeted layers in neural networks.<n>We show that uniformly applying dropout across layers does not consistently improve performance.<n>We also show that a stop-gradient operation is critical for ensuring dropout functions effectively as an augmentation.
  arXiv  Detail & Related papers  (2023-03-25T19:03:57Z)
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning [97.28695683236981]
  More gradient updates decrease the expressivity of the current value network. We demonstrate this phenomenon on Atari and Gym benchmarks, in both offline and online RL settings.
  arXiv  Detail & Related papers  (2020-10-27T17:55:16Z)
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation [53.8171136907856]
  We introduce a set of simple yet effective data augmentation strategies dubbed cutoff. cutoff relies on sampling consistency and thus adds little computational overhead. cutoff consistently outperforms adversarial training and achieves state-of-the-art results on the IWSLT2014 German-English dataset.
  arXiv  Detail & Related papers  (2020-09-29T07:08:35Z)
• A Theoretical Framework for Target Propagation [75.52598682467817]
  We analyze target propagation (TP), a popular but not yet fully understood alternative to backpropagation (BP) Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training.
  arXiv  Detail & Related papers  (2020-06-25T12:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.