An Empirical Study of Feature Selection Granularity

• URL: http://arxiv.org/abs/2607.24145v1
• Date: Mon, 27 Jul 2026 08:24:40 GMT
• Title: An Empirical Study of Feature Selection Granularity
• Abstract summary: Conventional feature selection algorithms compute feature importance scores globally across the entire feature set and then select the top-ranked features in a single step.<n>This approach raises a critical question: Can the presence of less informative (or noisy) features mask or obscure the true importance of other, more relevant features?
• Score: 3.3545793786322347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Feature selection aims to identify the most informative and relevant features for a given dataset, either in terms of capturing the underlying data structure and distribution better, or with respect to the performance on a downstream task. Existing research in this area has largely focused on developing novel algorithms (in both supervised and unsupervised settings), proposing new evaluation metrics and frameworks, or benchmarking the performance of existing methods. In this work, we examine feature selection through an algorithmic design perspective. Conventional feature selection algorithms typically compute feature importance scores globally across the entire feature set and then select the top-ranked features in a single step. However, this approach raises a critical question: Can the presence of less informative (or noisy) features mask or obscure the true importance of other, more relevant features? In other words, would a recursive strategy, where features are removed one by one while re-evaluating importance at each step, yield different and potentially better results than the standard global ranking approach? To answer this question, we conduct an extensive empirical study using five diverse feature selection algorithms. We implement each algorithm under both the conventional global selection design and the greedy recursive elimination design. We then analyze the impact of this algorithmic choice, both individually for each method and collectively across all methods, on a range of standard feature selection evaluation metrics. The empirical evaluation results show that the greedy approach improves the overall feature selection quality almost consistently, albeit on the expense of higher computational cost, supporting our initial expectation that the curse of dimensionality also obscures the ways of mitigating it.

Related papers

• Nearly Optimal Active Preference Learning and Its Application to LLM Alignment [68.56793807995417]
  Aligning large language models depends on high-quality datasets of human preference labels.<n>Many existing approaches adopt classical experimental design criteria such as G- or D-optimality.<n>In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives.
  arXiv  Detail & Related papers  (2026-02-02T03:21:29Z)
• Cascaded two-stage feature clustering and selection via separability and consistency in fuzzy decision systems [8.048511956662336]
  Feature selection is a vital technique in machine learning, as it can reduce computational complexity, improve model performance, and mitigate the risk of overfitting. This paper proposes a cascaded two-stage feature clustering and selection algorithm for fuzzy decision systems. The effectiveness of our proposed algorithm is evaluated through experiments conducted on 18 public datasets and a real-world schizophrenia dataset.
  arXiv  Detail & Related papers  (2024-07-22T02:44:32Z)
• Feature Selection as Deep Sequential Generative Learning [50.00973409680637]
  We develop a deep variational transformer model over a joint of sequential reconstruction, variational, and performance evaluator losses. Our model can distill feature selection knowledge and learn a continuous embedding space to map feature selection decision sequences into embedding vectors associated with utility scores.
  arXiv  Detail & Related papers  (2024-03-06T16:31:56Z)
• Sequential Attention for Feature Selection [12.89764845700709]
  We propose a feature selection algorithm called Sequential Attention that achieves state-of-the-art empirical results for neural networks. We give theoretical insights into our algorithm for linear regression by showing that an adaptation to this setting is equivalent to the classical Orthogonal Matching Pursuit (OMP) algorithm.
  arXiv  Detail & Related papers  (2022-09-29T15:49:06Z)
• Compactness Score: A Fast Filter Method for Unsupervised Feature Selection [66.84571085643928]
  We propose a fast unsupervised feature selection method, named as, Compactness Score (CSUFS) to select desired features. Our proposed algorithm seems to be more accurate and efficient compared with existing algorithms.
  arXiv  Detail & Related papers  (2022-01-31T13:01:37Z)
• Estimating leverage scores via rank revealing methods and randomization [50.591267188664666]
  We study algorithms for estimating the statistical leverage scores of rectangular dense or sparse matrices of arbitrary rank. Our approach is based on combining rank revealing methods with compositions of dense and sparse randomized dimensionality reduction transforms.
  arXiv  Detail & Related papers  (2021-05-23T19:21:55Z)
• Algorithmic Stability and Generalization of an Unsupervised Feature Selection Algorithm [20.564573628659918]
  Algorithmic stability is a key characteristic of an algorithm regarding its sensitivity to perturbations of input samples. In this paper, we propose an innovative unsupervised feature selection algorithm attaining this stability with provable guarantees.
  arXiv  Detail & Related papers  (2020-10-19T12:25:39Z)
• A Novel Community Detection Based Genetic Algorithm for Feature Selection [3.8848561367220276]
  Authors propose a genetic algorithm based on community detection, which functions in three steps. Nine benchmark classification problems were analyzed in terms of the performance of the presented approach.
  arXiv  Detail & Related papers  (2020-08-08T15:39:30Z)
• Run2Survive: A Decision-theoretic Approach to Algorithm Selection based on Survival Analysis [75.64261155172856]
  survival analysis (SA) naturally supports censored data and offers appropriate ways to use such data for learning distributional models of algorithm runtime. We leverage such models as a basis of a sophisticated decision-theoretic approach to algorithm selection, which we dub Run2Survive. In an extensive experimental study with the standard benchmark ASlib, our approach is shown to be highly competitive and in many cases even superior to state-of-the-art AS approaches.
  arXiv  Detail & Related papers  (2020-07-06T15:20:17Z)
• On-the-Fly Joint Feature Selection and Classification [16.84451472788859]
  We propose a framework to perform joint feature selection and classification on-the-fly. We derive the optimum solution of the associated optimization problem and analyze its structure. We evaluate the performance of the proposed algorithms on several public datasets.
  arXiv  Detail & Related papers  (2020-04-21T19:19:39Z)
• Extreme Algorithm Selection With Dyadic Feature Representation [78.13985819417974]
  We propose the setting of extreme algorithm selection (XAS) where we consider fixed sets of thousands of candidate algorithms. We assess the applicability of state-of-the-art AS techniques to the XAS setting and propose approaches leveraging a dyadic feature representation.
  arXiv  Detail & Related papers  (2020-01-29T09:40:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.