Reducing Learner Redundancy in Boosting via Residual Orthogonalization

• URL: http://arxiv.org/abs/2606.17567v1
• Date: Tue, 16 Jun 2026 06:13:59 GMT
• Title: Reducing Learner Redundancy in Boosting via Residual Orthogonalization
• Abstract summary: Our framework tackles redundancy through two complementary mechanisms: Spectral Residual Projection (SRP) and Covariance-Regularized Weighting (CRW)<n>Experiments across ten benchmark datasets demonstrate that SCBoost delivers strong out-of-the-box performance.<n>This work reinterprets boosting through a geometric lens, suggesting that explicit redundancy control is a principled and necessary step toward more efficient ensemble architectures.
• Score: 18.700465769971434
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While sequential residual fitting is the bedrock of standard boosting frameworks, it inherently breeds learner redundancy by repeatedly revisiting correlated error components. To address this bottleneck, we propose a shift from residual fitting to \textit{residual orthogonalization} and introduce SCBoost. Our framework tackles redundancy through two complementary mechanisms: Spectral Residual Projection (SRP) and Covariance-Regularized Weighting (CRW). During training, SRP projects each residual target onto the orthogonal complement of the historical prediction subspace, forcing successive learners to capture only novel empirical innovations. During aggregation, CRW optimizes ensemble weights on a validation set with an explicit covariance penalty to mitigate remaining correlations. Theoretically, we provide a finite-sample geometric characterization proving that SRP yields an exact additive residual-energy decomposition. Furthermore, under an isotropic-noise assumption, we rigorously establish the conditions under which this projection improves the effective Signal-to-Noise Ratio. Extensive experiments across ten benchmark datasets demonstrate that SCBoost delivers strong out-of-the-box performance, particularly in accuracy and F1 score. This work reinterprets boosting through a geometric lens, suggesting that explicit redundancy control is a principled and necessary step toward more efficient ensemble architectures.

Related papers

• SATB-VR: Training Few-Step Video Restoration Diffusion Model using SNR-Aware Trajectory Blending [55.764018145256216]
  We present SATB-VR, a few-step paradigm that jump-starts the denoising process via an auxiliary predictor.<n>We show that SATB-VR performs favorably against existing approaches on synthetic, real-world, and AIGC benchmarks.
  arXiv  Detail & Related papers  (2026-06-27T01:32:16Z)
• Residual-Controlled Multiplier Learning for Stochastic Constrained Decision-Making [12.04727482907937]
  CML addresses the problem of tracking the noise of primal feedback multipliers under mini-batch conditions.<n>We decompose the multiplier into an effective pressure signal for primal descent and a finite-gain multiplier.
  arXiv  Detail & Related papers  (2026-06-05T09:35:22Z)
• Supervised Distributional Reduction via Optimal Transport and Dependence Maximization [1.0499611180329804]
  We propose Supervised Distribution Reduction (SDR), an algorithm for learning target-aware representations.<n>SDR builds on the Fused Gromov-Wasserstein (FGW) objective to align the relational structure of the input distribution with a set of representative points.<n>SDR naturally induces a data-dependent, non-stationary geometry that can be leveraged for settings such as Gaussian Process (GP) modelling.
  arXiv  Detail & Related papers  (2026-05-26T19:38:20Z)
• Energy-Structured Low-Rank Adaptation for Continual Learning [78.5390284258307]
  We propose textbfEnergy-Concentrated and textbfEnergy-Ordered textbfLow-textbfRank textbfAdaptation (E$2$-LoRA)<n>By explicitly ordering and concentrating knowledge into leading ranks, E$2$-LoRA frees capacity for subsequent tasks.
  arXiv  Detail & Related papers  (2026-05-26T12:55:02Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• HIR-ALIGN: Enhancing Hyperspectral Image Restoration via Diffusion-Based Data Generation [50.46910397782266]
  Hyperspectral image (HSI) restoration is crucial for reliable analysis, as real HSIs suffer from degradations like noise, blur, and resolution loss.<n>We present HIR-ALIGN, a plug-and-play target-adaptive augmentation framework that enhances hyperspectral image restoration by augmenting limited training images with synthetic data that closely matches the target distribution using no extra data.
  arXiv  Detail & Related papers  (2026-05-13T14:14:13Z)
• Robust Low-Rank Tensor Completion based on M-product with Weighted Correlated Total Variation and Sparse Regularization [0.0]
  We introduce a weighted correlated total variation (TWCTV) regularizer that combines a weighted Schatten-$p$ norm on gradient tensors for low-rankness with smoothness enforcement and weighted sparse components for noise suppression.<n>The proposed weighting scheme adaptively reduces the thresholding level to preserve both dominant singular values and sparse components.<n>Through a systematic algorithmic approach, we introduce an enhanced alternating direction method of multipliers (ADMM) that offers both computational efficiency and theoretical substantiation.
  arXiv  Detail & Related papers  (2026-04-15T06:20:28Z)
• Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data [77.19117648492104]
  We present Federated Robust Curvature Optimization (FedRCO), a novel second-order optimization framework designed to improve convergence speed and reduce communication cost.<n>FedRCO addresses these challenges by integrating an efficient approximate curvature with a provable stability mechanism.<n>We show that FedRCO achieves superior robustness against diverse non-IID scenarios while achieving higher accuracy and faster convergence than both state-of-the-art first-order and second-order methods.
  arXiv  Detail & Related papers  (2026-03-30T11:37:46Z)
• Observationally Informed Adaptive Causal Experimental Design [55.998153710215654]
  We propose Active Residual Learning, a new paradigm that leverages the observational model as a foundational prior.<n>This approach shifts the experimental focus from learning target causal quantities from scratch to efficiently estimating the residuals required to correct observational bias.<n> Experiments on synthetic and semi-synthetic benchmarks demonstrate that R-Design significantly outperforms baselines.
  arXiv  Detail & Related papers  (2026-03-04T06:52:37Z)
• PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training [5.275001711555517]
  We propose Principal-Random Subspace for LLM Activation Compression (PRAC)<n>PRAC decomposes activations into two components: a principal subspace captured via SVD to retain dominant information, and a random subspace sampled from the orthogonal complement to approximate the tail.<n>Experiments on pre-training and fine-tuning tasks demonstrate that PRAC achieves up to 36% total memory reduction with negligible performance degradation and minimal computational cost.
  arXiv  Detail & Related papers  (2026-02-26T15:23:34Z)
• Evaluating the Efficiency of Latent Spaces via the Coupling-Matrix [0.5013248430919224]
  We introduce a redundancy index, denoted rho(C), that directly quantifies inter-dimensional dependencies.<n>Low rho(C) reliably predicts high classification accuracy or low reconstruction error, while elevated redundancy is associated with performance collapse.<n>We show that Tree-structured Parzen Estimators (TPE) preferentially explore low-rho regions, suggesting that rho(C) can guide neural architecture search and serve as a redundancy-aware regularization target.
  arXiv  Detail & Related papers  (2025-09-08T03:36:47Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.