A Notion of Feature Importance by Decorrelation and Detection of Trends by Random Forest Regression

• URL: http://arxiv.org/abs/2303.01156v1
• Date: Thu, 2 Mar 2023 11:01:49 GMT
• Title: A Notion of Feature Importance by Decorrelation and Detection of Trends by Random Forest Regression
• Authors: Yannick Gerstorfer, Lena Krieg, Max Hahn-Klimroth
• Abstract summary: We introduce a novel notion of feature importance based on the well-studied Gram-Schmidt decorrelation method. We propose two estimators for identifying trends in the data using random forest regression.
• Score: 1.675857332621569
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many studies, we want to determine the influence of certain features on a dependent variable. More specifically, we are interested in the strength of the influence -- i.e., is the feature relevant? -- and, if so, how the feature influences the dependent variable. Recently, data-driven approaches such as \emph{random forest regression} have found their way into applications (Boulesteix et al., 2012). These models allow to directly derive measures of feature importance, which are a natural indicator of the strength of the influence. For the relevant features, the correlation or rank correlation between the feature and the dependent variable has typically been used to determine the nature of the influence. More recent methods, some of which can also measure interactions between features, are based on a modeling approach. In particular, when machine learning models are used, SHAP scores are a recent and prominent method to determine these trends (Lundberg et al., 2017). In this paper, we introduce a novel notion of feature importance based on the well-studied Gram-Schmidt decorrelation method. Furthermore, we propose two estimators for identifying trends in the data using random forest regression, the so-called absolute and relative transversal rate. We empirically compare the properties of our estimators with those of well-established estimators on a variety of synthetic and real-world datasets.

Related papers

• Challenges in Variable Importance Ranking Under Correlation [6.718144470265263]
  We present a comprehensive simulation study investigating the impact of feature correlation on the assessment of variable importance. While there is always no correlation between knockoff variables and its corresponding predictor variables, we prove that the correlation increases linearly beyond a certain correlation threshold between the predictor variables.
  arXiv  Detail & Related papers  (2024-02-05T19:02:13Z)
• Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies [58.179981892921056]
  This work introduces a novel principle for disentanglement we call mechanism sparsity regularization. We propose a representation learning method that induces disentanglement by simultaneously learning the latent factors. We show that the latent factors can be recovered by regularizing the learned causal graph to be sparse.
  arXiv  Detail & Related papers  (2024-01-10T02:38:21Z)
• Identifiable Latent Polynomial Causal Models Through the Lens of Change [82.14087963690561]
  Causal representation learning aims to unveil latent high-level causal representations from observed low-level data. One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as identifiability.
  arXiv  Detail & Related papers  (2023-10-24T07:46:10Z)
• Causal Feature Selection via Transfer Entropy [59.999594949050596]
  Causal discovery aims to identify causal relationships between features with observational data. We introduce a new causal feature selection approach that relies on the forward and backward feature selection procedures. We provide theoretical guarantees on the regression and classification errors for both the exact and the finite-sample cases.
  arXiv  Detail & Related papers  (2023-10-17T08:04:45Z)
• Stubborn Lexical Bias in Data and Models [50.79738900885665]
  We use a new statistical method to examine whether spurious patterns in data appear in models trained on the data. We apply an optimization approach to *reweight* the training data, reducing thousands of spurious correlations. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models.
  arXiv  Detail & Related papers  (2023-06-03T20:12:27Z)
• Opening the random forest black box by the analysis of the mutual impact of features [0.0]
  We propose two novel approaches that focus on the mutual impact of features in random forests. MFI and MIR are very promising to shed light on the complex relationships between features and outcome.
  arXiv  Detail & Related papers  (2023-04-05T15:03:46Z)
• Data-Driven Influence Functions for Optimization-Based Causal Inference [105.5385525290466]
  We study a constructive algorithm that approximates Gateaux derivatives for statistical functionals by finite differencing. We study the case where probability distributions are not known a priori but need to be estimated from data.
  arXiv  Detail & Related papers  (2022-08-29T16:16:22Z)
• Triplot: model agnostic measures and visualisations for variable importance in predictive models that take into account the hierarchical correlation structure [3.0036519884678894]
  We propose new methods to support model analysis by exploiting the information about the correlation between variables. We show how to analyze groups of variables (aspects) both when they are proposed by the user and when they should be determined automatically. We also present the new type of model visualisation, triplot, which exploits a hierarchical structure of variable grouping to produce a high information density model visualisation.
  arXiv  Detail & Related papers  (2021-04-07T21:29:03Z)
• Estimating Causal Effects with the Neural Autoregressive Density Estimator [6.59529078336196]
  We use neural autoregressive density estimators to estimate causal effects within the Pearl's do-calculus framework. We show that the approach can retrieve causal effects from non-linear systems without explicitly modeling the interactions between the variables.
  arXiv  Detail & Related papers  (2020-08-17T13:12:38Z)
• On Disentangled Representations Learned From Correlated Data [59.41587388303554]
  We bridge the gap to real-world scenarios by analyzing the behavior of the most prominent disentanglement approaches on correlated data. We show that systematically induced correlations in the dataset are being learned and reflected in the latent representations. We also demonstrate how to resolve these latent correlations, either using weak supervision during training or by post-hoc correcting a pre-trained model with a small number of labels.
  arXiv  Detail & Related papers  (2020-06-14T12:47:34Z)
• Nonparametric Feature Impact and Importance [0.6123324869194193]
  We give mathematical definitions of feature impact and importance, derived from partial dependence curves, that operate directly on the data. To assess quality, we show that features ranked by these definitions are competitive with existing feature selection techniques.
  arXiv  Detail & Related papers  (2020-06-08T17:07:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.