Leveraging Model Inherent Variable Importance for Stable Online Feature Selection

• URL: http://arxiv.org/abs/2006.10398v1
• Date: Thu, 18 Jun 2020 10:01:18 GMT
• Title: Leveraging Model Inherent Variable Importance for Stable Online Feature Selection
• Authors: Johannes Haug, Martin Pawelczyk, Klaus Broelemann, Gjergji Kasneci
• Abstract summary: We introduce FIRES, a novel framework for online feature selection. Our framework is generic in that it leaves the choice of the underlying model to the user. Experiments show that the proposed framework is clearly superior in terms of feature selection stability.
• Score: 16.396739487911056
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Feature selection can be a crucial factor in obtaining robust and accurate predictions. Online feature selection models, however, operate under considerable restrictions; they need to efficiently extract salient input features based on a bounded set of observations, while enabling robust and accurate predictions. In this work, we introduce FIRES, a novel framework for online feature selection. The proposed feature weighting mechanism leverages the importance information inherent in the parameters of a predictive model. By treating model parameters as random variables, we can penalize features with high uncertainty and thus generate more stable feature sets. Our framework is generic in that it leaves the choice of the underlying model to the user. Strikingly, experiments suggest that the model complexity has only a minor effect on the discriminative power and stability of the selected feature sets. In fact, using a simple linear model, FIRES obtains feature sets that compete with state-of-the-art methods, while dramatically reducing computation time. In addition, experiments show that the proposed framework is clearly superior in terms of feature selection stability.

Related papers

• REFRESH: Responsible and Efficient Feature Reselection Guided by SHAP Values [17.489279048199304]
  REFRESH is a method to reselect features so that additional constraints that are desirable towards model performance can be achieved without having to train several new models. REFRESH's underlying algorithm is a novel technique using SHAP values and correlation analysis that can approximate for the predictions of a model without having to train these models.
  arXiv  Detail & Related papers  (2024-03-13T18:06:43Z)
• Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model [86.9619638550683]
  Vision-language foundation models have exhibited remarkable success across a multitude of downstream tasks due to their scalability on extensive image-text paired data. However, these models display significant limitations when applied to downstream tasks, such as fine-grained image classification, as a result of decision shortcuts''
  arXiv  Detail & Related papers  (2024-03-01T09:01:53Z)
• Causal Feature Selection via Transfer Entropy [59.999594949050596]
  Causal discovery aims to identify causal relationships between features with observational data. We introduce a new causal feature selection approach that relies on the forward and backward feature selection procedures. We provide theoretical guarantees on the regression and classification errors for both the exact and the finite-sample cases.
  arXiv  Detail & Related papers  (2023-10-17T08:04:45Z)
• Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers [66.36045164286854]
  We analyze a set of existing bias features and demonstrate there is no single model that works best for all the cases. By choosing an appropriate bias model, we can obtain a better robustness result than baselines with a more sophisticated model design.
  arXiv  Detail & Related papers  (2022-10-28T17:52:10Z)
• Ensembling improves stability and power of feature selection for deep learning models [11.973624420202388]
  In this paper, we show that inherentity in the design and training of deep learning models makes commonly used feature importance scores unstable. We explore the ensembling of feature importance scores of models across different epochs and find that this simple approach can substantially address this issue. We present a framework to combine the feature importance of trained models and instead of selecting features from one best model, we perform an ensemble of feature importance scores from numerous good models.
  arXiv  Detail & Related papers  (2022-10-02T19:07:53Z)
• Error-based Knockoffs Inference for Controlled Feature Selection [49.99321384855201]
  We propose an error-based knockoff inference method by integrating the knockoff features, the error-based feature importance statistics, and the stepdown procedure together. The proposed inference procedure does not require specifying a regression model and can handle feature selection with theoretical guarantees.
  arXiv  Detail & Related papers  (2022-03-09T01:55:59Z)
• Employing an Adjusted Stability Measure for Multi-Criteria Model Fitting on Data Sets with Similar Features [0.1127980896956825]
  We show that our approach achieves the same or better predictive performance compared to the two established approaches. Our approach succeeds at selecting the relevant features while avoiding irrelevant or redundant features. For data sets with many similar features, the feature selection stability must be evaluated with an adjusted stability measure.
  arXiv  Detail & Related papers  (2021-06-15T12:48:07Z)
• Modeling the Second Player in Distributionally Robust Optimization [90.25995710696425]
  We argue for the use of neural generative models to characterize the worst-case distribution. This approach poses a number of implementation and optimization challenges. We find that the proposed approach yields models that are more robust than comparable baselines.
  arXiv  Detail & Related papers  (2021-03-18T14:26:26Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)
• Feature Selection for Huge Data via Minipatch Learning [0.0]
  We propose Stable Minipatch Selection (STAMPS) and Adaptive STAMPS. STAMPS are meta-algorithms that build ensembles of selection events of base feature selectors trained on tiny, (ly-adaptive) random subsets of both the observations and features of the data. Our approaches are general and can be employed with a variety of existing feature selection strategies and machine learning techniques.
  arXiv  Detail & Related papers  (2020-10-16T17:41:08Z)
• Better Model Selection with a new Definition of Feature Importance [8.914907178577476]
  Feature importance aims at measuring how crucial each input feature is for model prediction. In this paper, we propose a new tree-model explanation approach for model selection.
  arXiv  Detail & Related papers  (2020-09-16T14:32:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.