Correcting Variable Importance Scored by Random Forests

• URL: http://arxiv.org/abs/2606.10770v1
• Date: Tue, 09 Jun 2026 12:25:51 GMT
• Title: Correcting Variable Importance Scored by Random Forests
• Abstract summary: The calculation of variable importance in Random Forests (RF) does not take into account the correlations among variables.<n>We propose to group variables by their conditional correlations (conditional on the response variable)<n>Our experiments show that both lead to sensible corrections to the importance of variables.
• Score: 2.3690789353712405
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Variable importance produced by Random Forests (RF) is used widely in statistical data analysis, and has played an important role in a variety of tasks such as assisting model interpretation, model selection and diagnosis, and cost-bounded learning etc. However, the calculation of variable importance in RF does not take into account of the correlations among variables, and variables that are correlated to many other variables tend to receive a lower importance index or being completely masked (i.e., with an importance index near zero) by other strongly correlated variables. To prevent influence from unwanted correlated variables in calculating variable importance, we propose to group variables by their conditional correlations (conditional on the response variable). We explore two computationally efficient options, with one grouping variables individually, and then separates the variable of interest from all correlated variables, while the other uses clustering to group variables according to their pair-wise conditional correlations. Our experiments show that both lead to sensible corrections to the importance of variables.

Related papers

• Temporal Latent Variable Structural Causal Model for Causal Discovery under External Interferences [53.308122815325326]
  We introduce latent variables to represent unobserved factors that affect the observed data.<n>Specifically, to capture the causal strength and adjacency information, we propose a new temporal latent variable structural causal model.<n>Considering that expert knowledge can provide information about unknown interferences in certain scenarios, we develop a method that facilitates the incorporation of prior knowledge into parameter learning.
  arXiv  Detail & Related papers  (2025-11-13T07:10:10Z)
• Local Learning for Covariate Selection in Nonparametric Causal Effect Estimation with Latent Variables [17.154150811381907]
  Estimating causal effects from nonexperimental data is a fundamental problem in many fields of science.<n>We propose a novel local learning approach for covariate selection in nonparametric causal effect estimation.<n>We validate our algorithm through extensive experiments on both synthetic and real-world data.
  arXiv  Detail & Related papers  (2024-11-25T12:08:54Z)
• Challenges in Variable Importance Ranking Under Correlation [6.718144470265263]
  We present a comprehensive simulation study investigating the impact of feature correlation on the assessment of variable importance. While there is always no correlation between knockoff variables and its corresponding predictor variables, we prove that the correlation increases linearly beyond a certain correlation threshold between the predictor variables.
  arXiv  Detail & Related papers  (2024-02-05T19:02:13Z)
• Gower's similarity coefficients with automatic weight selection [0.0]
  The most popular dissimilarity for mixed-type variables is derived as the complement to one of the Gower's similarity coefficient. The discussion on the weighting schemes is sometimes misleading since it often ignores that the unweighted "standard" setting hides an unbalanced contribution of the single variables to the overall dissimilarity. We address this drawback following the recent idea of introducing a weighting scheme that minimizes the differences in the correlation between each contributing dissimilarity and the resulting weighted Gower's dissimilarity.
  arXiv  Detail & Related papers  (2024-01-30T14:21:56Z)
• A Causal Framework for Decomposing Spurious Variations [68.12191782657437]
  We develop tools for decomposing spurious variations in Markovian and Semi-Markovian models. We prove the first results that allow a non-parametric decomposition of spurious effects. The described approach has several applications, ranging from explainable and fair AI to questions in epidemiology and medicine.
  arXiv  Detail & Related papers  (2023-06-08T09:40:28Z)
• Identifying Weight-Variant Latent Causal Models [82.14087963690561]
  We find that transitivity acts as a key role in impeding the identifiability of latent causal representations. Under some mild assumptions, we can show that the latent causal representations can be identified up to trivial permutation and scaling. We propose a novel method, termed Structural caUsAl Variational autoEncoder, which directly learns latent causal representations and causal relationships among them.
  arXiv  Detail & Related papers  (2022-08-30T11:12:59Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets [53.34152466646884]
  In this paper, we show how bringing recent results on equivariant representation learning instantiated on structured spaces together with simple use of classical results on causal inference provides an effective practical solution. We demonstrate how our model allows dealing with more than one nuisance variable under some assumptions and can enable analysis of pooled scientific datasets in scenarios that would otherwise entail removing a large portion of the samples.
  arXiv  Detail & Related papers  (2022-03-29T04:54:06Z)
• VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning [84.70916463298109]
  VarCLR is a new approach for learning semantic representations of variable names. VarCLR is an excellent fit for contrastive learning, which aims to minimize the distance between explicitly similar inputs. We show that VarCLR enables the effective application of sophisticated, general-purpose language models like BERT.
  arXiv  Detail & Related papers  (2021-12-05T18:40:32Z)
• Obtaining Causal Information by Merging Datasets with MAXENT [12.64433334351049]
  We discuss how causal knowledge can be obtained without having observed all variables jointly. We derive bounds on the interventional distribution and the average causal effect of a treatment on a target variable in the presence of confounders.
  arXiv  Detail & Related papers  (2021-07-15T23:16:36Z)
• Stable Prediction via Leveraging Seed Variable [73.9770220107874]
  Previous machine learning methods might exploit subtly spurious correlations in training data induced by non-causal variables for prediction. We propose a conditional independence test based algorithm to separate causal variables with a seed variable as priori, and adopt them for stable prediction. Our algorithm outperforms state-of-the-art methods for stable prediction.
  arXiv  Detail & Related papers  (2020-06-09T06:56:31Z)
• Be Like Water: Robustness to Extraneous Variables Via Adaptive Feature Normalization [17.829013101192295]
  Extraneous variables are variables that are irrelevant for a certain task, but heavily affect the distribution of the available data. We show that the presence of such variables can degrade the performance of deep-learning models. We show that estimating the feature statistics adaptively during inference, as in instance normalization, addresses this issue.
  arXiv  Detail & Related papers  (2020-02-10T18:47:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.