How Ensemble Learning Balances Accuracy and Overfitting: A Bias-Variance Perspective on Tabular Data

• URL: http://arxiv.org/abs/2512.05469v1
• Date: Fri, 05 Dec 2025 06:51:06 GMT
• Title: How Ensemble Learning Balances Accuracy and Overfitting: A Bias-Variance Perspective on Tabular Data
• Abstract summary: This study examines how ensembles balance accuracy and overfitting across four classification tasks.<n>The results show that ensembles can reach high accuracy without large gaps by reducing variance through averaging or controlled boosting.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensemble models often achieve higher accuracy than single learners, but their ability to maintain small generalization gaps is not always well understood. This study examines how ensembles balance accuracy and overfitting across four tabular classification tasks: Breast Cancer, Heart Disease, Pima Diabetes, and Credit Card Fraud. Using repeated stratified cross validation with statistical significance testing, we compare linear models, a single decision tree, and nine ensemble methods. The results show that ensembles can reach high accuracy without large gaps by reducing variance through averaging or controlled boosting. On nearly linear and clean data, linear models already generalize well and ensembles offer little additional benefit. On datasets with meaningful nonlinear structure, tree based ensembles increase test accuracy by 5 to 7 points while keeping gaps below 3 percent. On noisy or highly imbalanced datasets, ensembles remain competitive but require regularization to avoid fitting noise or majority class patterns. We also compute simple dataset complexity indicators, such as linearity score, Fisher ratio, and noise estimate, which explain when ensembles are likely to control variance effectively. Overall, the study provides a clear view of how and when ensembles maintain high accuracy while keeping overfitting low, offering practical guidance for model selection in real world tabular applications.

Related papers

• Unique Rashomon Sets for Robust Active Learning [0.0]
  We introduce UNique Rashomon Ensembled Active Learning (UNREAL)<n>UNREAL selectively ensembles models from the Rashomon set, which is the set of nearly optimal models.<n>We show that UNREAL achieves faster theoretical convergence rates than traditional active learning approaches.
  arXiv  Detail & Related papers  (2025-03-09T20:50:34Z)
• Probably Approximately Precision and Recall Learning [60.00180898830079]
  A key challenge in machine learning is the prevalence of one-sided feedback.<n>We introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels.<n>We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• Conformal-in-the-Loop for Learning with Imbalanced Noisy Data [5.69777817429044]
  Class imbalance and label noise are pervasive in large-scale datasets.<n>Much of machine learning research assumes well-labeled, balanced data, which rarely reflects real world conditions.<n>We propose Conformal-in-the-Loop (CitL), a novel training framework that addresses both challenges with a conformal prediction-based approach.
  arXiv  Detail & Related papers  (2024-11-04T17:09:58Z)
• Reviving Undersampling for Long-Tailed Learning [16.054442161144603]
  We aim to enhance the accuracy of the worst-performing categories and utilize the harmonic mean and geometric mean to assess the model's performance. We devise a straightforward model ensemble strategy, which does not result in any additional overhead and achieves improved harmonic and geometric mean. We validate the effectiveness of our approach on widely utilized benchmark datasets for long-tailed learning.
  arXiv  Detail & Related papers  (2024-01-30T08:15:13Z)
• Leveraging Linear Independence of Component Classifiers: Optimizing Size and Prediction Accuracy for Online Ensembles [3.97048491084787]
  We introduce a novel perspective, rooted in the linear independence of classifier's votes, to analyze the interplay between ensemble size and prediction accuracy. We present a method to determine the minimum ensemble size required to ensure a target probability of linearly independent votes. Surprisingly, the calculated ideal ensemble size deviates from empirical results for certain datasets, emphasizing the influence of other factors.
  arXiv  Detail & Related papers  (2023-08-27T18:38:09Z)
• Deep Negative Correlation Classification [82.45045814842595]
  Existing deep ensemble methods naively train many different models and then aggregate their predictions. We propose deep negative correlation classification (DNCC) DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
  arXiv  Detail & Related papers  (2022-12-14T07:35:20Z)
• Relieving Long-tailed Instance Segmentation via Pairwise Class Balance [85.53585498649252]
  Long-tailed instance segmentation is a challenging task due to the extreme imbalance of training samples among classes. It causes severe biases of the head classes (with majority samples) against the tailed ones. We propose a novel Pairwise Class Balance (PCB) method, built upon a confusion matrix which is updated during training to accumulate the ongoing prediction preferences.
  arXiv  Detail & Related papers  (2022-01-08T07:48:36Z)
• Simple data balancing achieves competitive worst-group-accuracy [16.197105634385693]
  We compare state-of-the-art methods to simple balancing of classes and groups by either subsampling or reweighting data. Our results show that these data balancing baselines achieve state-of-the-art-accuracy. Access to group information is most critical for model selection purposes, and not so much during training.
  arXiv  Detail & Related papers  (2021-10-27T15:15:11Z)
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss [72.62029620566925]
  Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm. Our work analyzes contrastive learning without assuming conditional independence of positive pairs. We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective.
  arXiv  Detail & Related papers  (2021-06-08T07:41:02Z)
• Selective Classification Can Magnify Disparities Across Groups [89.14499988774985]
  We find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities. Increasing abstentions can even decrease accuracies on some groups. We train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group.
  arXiv  Detail & Related papers  (2020-10-27T08:51:30Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.