There is no Double-Descent in Random Forests

• URL: http://arxiv.org/abs/2111.04409v1
• Date: Mon, 8 Nov 2021 12:25:32 GMT
• Title: There is no Double-Descent in Random Forests
• Abstract summary: Random Forests (RFs) are among the state-of-the-art in machine learning and offer excellent performance with nearly zero parameter tuning. Recently, a broadly received study argued that a RF exhibits a so-called double-descent curve: First, the model overfits the data in a u-shaped curve and then, once a certain model complexity is reached, it suddenly improves its performance again. In this paper, we challenge the notion that model capacity is the correct tool to explain the success of RF and argue that the algorithm which trains the model plays a more important role than previously thought
• Score: 6.967385165474138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Random Forests (RFs) are among the state-of-the-art in machine learning and offer excellent performance with nearly zero parameter tuning. Remarkably, RFs seem to be impervious to overfitting even though their basic building blocks are well-known to overfit. Recently, a broadly received study argued that a RF exhibits a so-called double-descent curve: First, the model overfits the data in a u-shaped curve and then, once a certain model complexity is reached, it suddenly improves its performance again. In this paper, we challenge the notion that model capacity is the correct tool to explain the success of RF and argue that the algorithm which trains the model plays a more important role than previously thought. We show that a RF does not exhibit a double-descent curve but rather has a single descent. Hence, it does not overfit in the classic sense. We further present a RF variation that also does not overfit although its decision boundary approximates that of an overfitted DT. Similar, we show that a DT which approximates the decision boundary of a RF will still overfit. Last, we study the diversity of an ensemble as a tool the estimate its performance. To do so, we introduce Negative Correlation Forest (NCForest) which allows for precise control over the diversity in the ensemble. We show, that the diversity and the bias indeed have a crucial impact on the performance of the RF. Having too low diversity collapses the performance of the RF into a a single tree, whereas having too much diversity means that most trees do not produce correct outputs anymore. However, in-between these two extremes we find a large range of different trade-offs with all roughly equal performance. Hence, the specific trade-off between bias and diversity does not matter as long as the algorithm reaches this good trade-off regime.

Related papers

• DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving [65.7087560656003]
  Generative diffusion models for end-to-end autonomous driving often suffer from mode collapse.<n>We propose DiffusionDriveV2, which leverages reinforcement learning to constrain low-quality modes and explore for superior trajectories.<n>This significantly enhances the overall output quality while preserving the inherent multimodality of its core Gaussian Mixture Model.
  arXiv  Detail & Related papers  (2025-12-08T17:29:52Z)
• Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning [23.202550214006237]
  It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to textitdiversity collapse, where outputs lack variety.<n>We introduce a principled method -- textitdifferential smoothing -- that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based languages.
  arXiv  Detail & Related papers  (2025-11-25T05:28:55Z)
• Think Twice: Branch-and-Rethink Reasoning Reward Model [32.70732791642558]
  We introduce branch-and-rethink (BR-RM), a two-turn RM that transfers the think-twice principle to reward modeling.<n>We train with GRPO-style reinforcement learning over structured two-turn traces using a simple binary outcome reward with strict format checks.<n>By converting all-at-oncescoringintofocused, second-lookreasoning, BR-RMreducesjudgmentdiffusionandimproves sensitivity to subtle yet consequential errors.
  arXiv  Detail & Related papers  (2025-10-27T17:58:07Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)
• Bias Fitting to Mitigate Length Bias of Reward Model in RLHF [81.44256822500257]
  Reinforcement Learning from Human Feedback relies on reward models to align large language models with human preferences.<n>We propose FiMi-RM, a framework that autonomously learns and corrects underlying bias patterns.<n> Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution.
  arXiv  Detail & Related papers  (2025-05-19T08:29:28Z)
• Robust Fine-tuning of Zero-shot Models via Variance Reduction [56.360865951192324]
  When fine-tuning zero-shot models, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD) We propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs.
  arXiv  Detail & Related papers  (2024-11-11T13:13:39Z)
• Heterogeneous Random Forest [2.0646127669654835]
  Heterogeneous Random Forest (HRF) is designed to enhance tree diversity in a meaningful way. HRF consistently outperformed other ensemble methods in terms of accuracy across the majority of datasets.
  arXiv  Detail & Related papers  (2024-10-24T09:18:55Z)
• On the Wasserstein Convergence and Straightness of Rectified Flow [54.580605276017096]
  Rectified Flow (RF) is a generative model that aims to learn straight flow trajectories from noise to data. We provide a theoretical analysis of the Wasserstein distance between the sampling distribution of RF and the target distribution. We present general conditions guaranteeing uniqueness and straightness of 1-RF, which is in line with previous empirical findings.
  arXiv  Detail & Related papers  (2024-10-19T02:36:11Z)
• More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatory [12.689249854199982]
  We show that the test risk of RF regression decreases monotonically with both the number of features and the number of samples. We then demonstrate that, for a large class of tasks characterized by powerlaw eigenstructure, training to near-zero training loss is obligatory.
  arXiv  Detail & Related papers  (2023-11-24T18:27:41Z)
• Self-NeRF: A Self-Training Pipeline for Few-Shot Neural Radiance Fields [17.725937326348994]
  We propose Self-NeRF, a self-evolved NeRF that iteratively refines the radiance fields with very few number of input views. In each iteration, we label unseen views with the predicted colors or warped pixels generated by the model from the preceding iteration. These expanded pseudo-views are afflicted by imprecision in color and warping artifacts, which degrades the performance of NeRF.
  arXiv  Detail & Related papers  (2023-03-10T08:22:36Z)
• Optimization Variance: Exploring Generalization Properties of DNNs [83.78477167211315]
  The test error of a deep neural network (DNN) often demonstrates double descent. We propose a novel metric, optimization variance (OV), to measure the diversity of model updates.
  arXiv  Detail & Related papers  (2021-06-03T09:34:17Z)
• Enhanced Principal Component Analysis under A Collaborative-Robust Framework [89.28334359066258]
  We introduce a general collaborative-robust weight learning framework that combines weight learning and robust loss in a non-trivial way. Under the proposed framework, only a part of well-fitting samples are activated which indicates more importance during training, and others, whose errors are large, will not be ignored. In particular, the negative effects of inactivated samples are alleviated by the robust loss function.
  arXiv  Detail & Related papers  (2021-03-22T15:17:37Z)
• Federated Multi-Armed Bandits [18.95281057580889]
  Federated multi-armed bandits (FMAB) is a new bandit paradigm that parallels the federated learning (FL) framework in supervised learning. This paper proposes a general framework of FMAB and then studies two specific federated bandit models. We show that, somewhat surprisingly, the order-optimal regret can be achieved independent of the number of clients with a careful choice of the update periodicity.
  arXiv  Detail & Related papers  (2021-01-28T18:59:19Z)
• Long-tailed Recognition by Routing Diverse Distribution-Aware Experts [64.71102030006422]
  We propose a new long-tailed classifier called RoutIng Diverse Experts (RIDE) It reduces the model variance with multiple experts, reduces the model bias with a distribution-aware diversity loss, reduces the computational cost with a dynamic expert routing module. RIDE outperforms the state-of-the-art by 5% to 7% on CIFAR100-LT, ImageNet-LT and iNaturalist 2018 benchmarks.
  arXiv  Detail & Related papers  (2020-10-05T06:53:44Z)
• Joints in Random Forests [13.096855747795303]
  Decision Trees (DTs) and Random Forests (RFs) are powerful discriminative learners and tools of central importance to the everyday machine learning practitioner and data scientist. We show that DTs and RFs can naturally be interpreted as generative models, by drawing a connection to Probabilistic Circuits. This reinterpretation equips them with a full joint distribution over the feature space and leads to Generative Decision Trees (GeDTs) and Generative Forests (GeFs)
  arXiv  Detail & Related papers  (2020-06-25T14:17:19Z)
• Fairness Through Robustness: Investigating Robustness Disparity in Deep Learning [61.93730166203915]
  We argue that traditional notions of fairness are not sufficient when the model is vulnerable to adversarial attacks. We show that measuring robustness bias is a challenging task for DNNs and propose two methods to measure this form of bias.
  arXiv  Detail & Related papers  (2020-06-17T22:22:24Z)
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference [119.19779637025444]
  Deep networks were recently suggested to face the odds between accuracy (on clean natural images) and robustness (on adversarially perturbed images) This paper studies multi-exit networks associated with input-adaptive inference, showing their strong promise in achieving a "sweet point" in cooptimizing model accuracy, robustness and efficiency.
  arXiv  Detail & Related papers  (2020-02-24T00:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.