Feature Bagging Provides Stability

• URL: http://arxiv.org/abs/2607.26964v2
• Date: Tue, 04 Aug 2026 00:17:57 GMT
• Title: Feature Bagging Provides Stability
• Abstract summary: We study feature bagging through the lens of algorithmic stability.<n>We introduce feature instability (FI), the feature-axis analogue of instance instability (II), which measures sensitivity to removing a single feature.<n>We show that a modest number of bagging rounds is sufficient to approach the infinite-bagging stability level.
• Score: 9.66778141968989
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study feature bagging through the lens of algorithmic stability. Feature bagging is an ensemble strategy that aggregates base learners trained on randomly subsampled feature subsets, possibly in a data-dependent manner. We introduce feature instability (FI), the feature-axis analogue of instance instability (II), which measures sensitivity to removing a single feature. Smaller values of II or FI correspond to stronger stability, and our experiments show that FI captures generalization-relevant information complementary to II. Within this framework, we analyze feature bagging in both a parametric linear model and a model-free setting inspired by recursive feature subsampling in random forests. In both settings, we establish formal guarantees showing that feature bagging improves the relevant stability relative to its non-bagged counterpart, with larger improvements under more aggressive subsampling. We further show that a modest number of bagging rounds is sufficient to approach the infinite-bagging stability level.

Related papers

• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Dual-Stage Invariant Continual Learning under Extreme Visual Sparsity [8.16821029459195]
  We show that background-driven gradients destabilize the feature backbone during sequential domain shifts.<n>We propose a dual-stage invariant continual learning framework via joint distillation.<n> Experiments on a high-resolution space-based RSO detection dataset show consistent improvement over established continual object detection methods.
  arXiv  Detail & Related papers  (2026-03-27T09:03:49Z)
• First-Mover Bias in Gradient Boosting Explanations: Mechanism, Detection, and Resolution [0.0]
  First-mover bias is a concentration of feature importance caused by sequential residual fitting in gradient boosting.<n>We show that model independence is sufficient to resolve first-mover bias in the linear regime, and remains the most effective mitigation under nonlinear data-generating processes.
  arXiv  Detail & Related papers  (2026-03-22T02:59:40Z)
• Causally-Guided Diffusion for Stable Feature Selection [29.398494058613124]
  We study feature selection from a stability perspective and introduce Causally-Guided Diffusion for Stable Feature Selection.<n>In CGDFS, we formalized feature selection as approximate posterior inference over feature subsets.<n>We evaluate CGDFS on open-source real-world datasets exhibiting distribution shifts.
  arXiv  Detail & Related papers  (2026-03-21T20:14:00Z)
• The Price of Robustness: Stable Classifiers Need Overparameterization [17.335490896384265]
  We establish a generalization bound for finite function classes that improves inversely with class stability.<n>We derive as a corollary a law of robustness for classification that extends the results of Bubeck and Sellke.<n>Experiments support our theory, while traditional norm-based measures remain largely uninformative.
  arXiv  Detail & Related papers  (2026-03-03T09:47:06Z)
• MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics [72.00014675808228]
  Instability in Large Language Models evaluation process obscures true learning dynamics.<n>We introduce textbfMaP, a framework that integrates underlineMerging underlineand the underlinePass@k metric.<n>Experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent rankings.
  arXiv  Detail & Related papers  (2025-10-10T11:40:27Z)
• Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performance [72.57668440744301]
  We propose a novel regularization that constrains the distance of fine-tuning and pre-trained model in the function space with simulated OOD samples.<n>Our approach could consistently improve both downstream task ID fine-tuning performance and OOD robustness across a variety of CLIP backbones.
  arXiv  Detail & Related papers  (2025-08-31T12:14:34Z)
• Stochastic Encodings for Active Feature Acquisition [100.47043816019888]
  Active Feature Acquisition is an instance-wise, sequential decision making problem.<n>The aim is to dynamically select which feature to measure based on current observations, independently for each test instance.<n>Common approaches either use Reinforcement Learning, which experiences training difficulties, or greedily maximize the conditional mutual information of the label and unobserved features, which makes myopic.<n>We introduce a latent variable model, trained in a supervised manner. Acquisitions are made by reasoning about the features across many possible unobserved realizations in a latent space.
  arXiv  Detail & Related papers  (2025-08-03T23:48:46Z)
• Stable Neighbor Denoising for Source-free Domain Adaptive Segmentation [91.83820250747935]
  Pseudo-label noise is mainly contained in unstable samples in which predictions of most pixels undergo significant variations during self-training. We introduce the Stable Neighbor Denoising (SND) approach, which effectively discovers highly correlated stable and unstable samples. SND consistently outperforms state-of-the-art methods in various SFUDA semantic segmentation settings.
  arXiv  Detail & Related papers  (2024-06-10T21:44:52Z)
• Bagging Provides Assumption-free Stability [11.456416081243654]
  Bagging is an important technique for stabilizing machine learning models. In this paper, we derive a finite-sample guarantee on the stability of bagging for any model.
  arXiv  Detail & Related papers  (2023-01-30T01:18:05Z)
• Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent [55.85456985750134]
  We introduce a new stability measure called on-average model stability, for which we develop novel bounds controlled by the risks of SGD iterates. This yields generalization bounds depending on the behavior of the best model, and leads to the first-ever-known fast bounds in the low-noise setting. To our best knowledge, this gives the firstever-known stability and generalization for SGD with even non-differentiable loss functions.
  arXiv  Detail & Related papers  (2020-06-15T06:30:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.