Bias in Filter Feature Selection Evaluation: A Meta-Analysis of Datasets, Baselines, and Experimental Design Choices

• URL: http://arxiv.org/abs/2606.07068v1
• Date: Fri, 05 Jun 2026 09:06:56 GMT
• Title: Bias in Filter Feature Selection Evaluation: A Meta-Analysis of Datasets, Baselines, and Experimental Design Choices
• Abstract summary: Recent developments in Deep Learning (DL) and data valuation in Machine Learning (ML) suggest that the evaluation of new methods may be consciously or unconsciously biased.<n>The aim of this study is therefore to examine FFS studies to identify factors that influence the evaluation and that might consist entry point for biases.<n>The analysis provides reflections on how to examine FFS studies, highlights lessons learned throughout the process, and gives five evidence-based recommendations for future FFS evaluation.
• Score: 0.22940141855172028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: Since 1990 many feature selection methods have been proposed across heterogeneous applications. To validate the usefulness of a new method, it needs to be compared against at least one baseline method from the existing literature on a feature selection task using at least one dataset. Recent developments in tabular Deep Learning (DL) and data valuation in Machine Learning (ML) suggest that the evaluation of new methods, algorithms, and models may be consciously or unconsciously biased. We hypothesise that a similar trend exists in feature selection (FS), particularly in filter feature selection (FFS). The aim of this study is therefore to examine FFS studies to identify factors that influence the evaluation and that might consist entry point for biases in order to recommend stronger principles for FFS evaluation. Methods: We analyse a sample of 28 high profile FFS studies published between 1994 and 2025. The analysis provides reflections on how to examine FFS studies, highlights lessons learned throughout the process, and gives five evidence-based recommendations for future FFS evaluation. Results: Multivariate Linear Regression analysis achieved a score of $R^2=0.33$. It means that 33% of the variance in the performance of new methods against chosen baselines (win rate) is explained by the number of datasets (#Datasets), the number of baselines (#Baselines), and the number of new methods (#NewMethods). Discussion: $R^2=0.33$ is considered medium explanation; which is promising given that this is the first such study. The medium explanation result is due to the fact that win rate is influenced by additional factors such as the maturity of the feature selection domain, the type of datasets and baselines, and the simplicity of the regression model used to explain the relationship.

Related papers

• Faithful and Fast Influence Function via Advanced Sampling [10.773767694482858]
  We propose two advanced sampling techniques based on features and logits.<n>These samplers select a small yet representative subset of the entire dataset by considering the distribution of features or logits.<n>We validate our approach through class removal experiments, using the F1-score to measure how effectively the model forgets a class.
  arXiv  Detail & Related papers  (2025-10-30T17:55:19Z)
• Data Selection for ERMs [67.57726352698933]
  We study how well can $mathcalA$ perform when trained on at most $nll N$ data points selected from a population of $N$ points.<n>Our results include optimal data-selection bounds for mean estimation, linear classification, and linear regression.
  arXiv  Detail & Related papers  (2025-04-20T11:26:01Z)
• Design Considerations in Offline Preference-based RL [25.683790623819952]
  We study how the different design choices made in methods such as DPO, IPO, SLiC and many variants influence the quality of the learned policy.<n>Our treatment yields insights into the choices of loss function, the policy which is used to normalize log-likelihoods, and also the role of the data sampling policy.
  arXiv  Detail & Related papers  (2025-02-08T00:01:37Z)
• Which Imputation Fits Which Feature Selection Method? A Survey-Based Simulation Study [4.335350817722218]
  Feature importance measures are usually considered for feature selection as well as to assess the effect of features on the outcome variables in the model.<n>The typical solution is to impute the missing data before applying the learning method.<n>We consider the two most common tree-based methods, Random Forest and XGBoost, and an interpretable linear model with regularization.
  arXiv  Detail & Related papers  (2024-12-18T07:36:03Z)
• Is Data Valuation Learnable and Interpretable? [3.9325957466009203]
  Current data valuation methods ignore the interpretability of the output values. This study aims to answer an important question: is data valuation learnable and interpretable?
  arXiv  Detail & Related papers  (2024-06-03T08:13:47Z)
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data [102.16105233826917]
  Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning.
  arXiv  Detail & Related papers  (2024-04-22T17:20:18Z)
• Continual Learning with Pre-Trained Models: A Survey [61.97613090666247]
  Continual Learning aims to overcome the catastrophic forgetting of former knowledge when learning new ones. This paper presents a comprehensive survey of the latest advancements in PTM-based CL.
  arXiv  Detail & Related papers  (2024-01-29T18:27:52Z)
• A data-science pipeline to enable the Interpretability of Many-Objective Feature Selection [0.1474723404975345]
  Many-Objective Feature Selection (MOFS) approaches use four or more objectives to determine the relevance of a subset of features in a supervised learning task. This paper proposes an original methodology to support data scientists in the interpretation and comparison of the MOFS outcome by combining post-processing and visualisation of the set of solutions.
  arXiv  Detail & Related papers  (2023-11-30T17:44:22Z)
• Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks [58.469818546042696]
  We study the sample efficiency of OPE with human preference and establish a statistical guarantee for it. By appropriately selecting the size of a ReLU network, we show that one can leverage any low-dimensional manifold structure in the Markov decision process.
  arXiv  Detail & Related papers  (2023-10-16T16:27:06Z)
• Selecting the suitable resampling strategy for imbalanced data classification regarding dataset properties [62.997667081978825]
  In many application domains such as medicine, information retrieval, cybersecurity, social media, etc., datasets used for inducing classification models often have an unequal distribution of the instances of each class. This situation, known as imbalanced data classification, causes low predictive performance for the minority class examples. Oversampling and undersampling techniques are well-known strategies to deal with this problem by balancing the number of examples of each class.
  arXiv  Detail & Related papers  (2021-12-15T18:56:39Z)
• Learning to Select Base Classes for Few-shot Classification [96.92372639495551]
  We use the Similarity Ratio as an indicator for the generalization performance of a few-shot model. We then formulate the base class selection problem as a submodular optimization problem over Similarity Ratio.
  arXiv  Detail & Related papers  (2020-04-01T09:55:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.