Statistical Test for Feature Selection Pipelines by Selective Inference

• URL: http://arxiv.org/abs/2406.18902v2
• Date: Sun, 13 Oct 2024 11:29:21 GMT
• Title: Statistical Test for Feature Selection Pipelines by Selective Inference
• Abstract summary: A data analysis pipeline is a structured sequence of steps that transforms raw data into meaningful insights. We propose a novel statistical test to assess the significance of data analysis pipelines in feature selection problems.
• Score: 13.628959580589665
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A data analysis pipeline is a structured sequence of steps that transforms raw data into meaningful insights by integrating various analysis algorithms. In this paper, we propose a novel statistical test to assess the significance of data analysis pipelines in feature selection problems. Our approach enables the systematic development of valid statistical tests applicable to any feature selection pipeline composed of predefined components. We develop this framework based on selective inference, a statistical technique that has recently gained attention for data-driven hypotheses. As a proof of concept, we consider feature selection pipelines for linear models, composed of three missing value imputation algorithms, three outlier detection algorithms, and three feature selection algorithms. We theoretically prove that our statistical test can control the probability of false positive feature selection at any desired level, and demonstrate its validity and effectiveness through experiments on synthetic and real data. Additionally, we present an implementation framework that facilitates testing across any configuration of these feature selection pipelines without extra implementation costs.

Related papers

• Statistical Testing Framework for Clustering Pipelines by Selective Inference [9.309321925569241]
  We propose a novel statistical testing framework to assess the significance of clustering results obtained through data analysis pipelines.<n>We prove that the proposed test controls the type I error rate at any nominal level and demonstrate its validity and effectiveness through experiments on synthetic and real datasets.
  arXiv  Detail & Related papers  (2026-03-19T02:20:35Z)
• Flexible Selective Inference with Flow-based Transport Maps [7.197592390105458]
  This paper introduces a new method that leverages tools from flow-based generative modeling to approximate a potentially complex conditional distribution.<n>We demonstrate that this method enables flexible selective inference by providing valid p-values and confidence sets for adaptively selected hypotheses and parameters.
  arXiv  Detail & Related papers  (2025-06-01T20:05:20Z)
• Meta-Statistical Learning: Supervised Learning of Statistical Inference [59.463430294611626]
  This work demonstrates that the tools and principles driving the success of large language models (LLMs) can be repurposed to tackle distribution-level tasks. We propose meta-statistical learning, a framework inspired by multi-instance learning that reformulates statistical inference tasks as supervised learning problems.
  arXiv  Detail & Related papers  (2025-02-17T18:04:39Z)
• A Hybrid Framework for Statistical Feature Selection and Image-Based Noise-Defect Detection [55.2480439325792]
  This paper presents a hybrid framework that integrates both statistical feature selection and classification techniques to improve defect detection accuracy. We present around 55 distinguished features that are extracted from industrial images, which are then analyzed using statistical methods. By integrating these methods with flexible machine learning applications, the proposed framework improves detection accuracy and reduces false positives and misclassifications.
  arXiv  Detail & Related papers  (2024-12-11T22:12:21Z)
• fastHDMI: Fast Mutual Information Estimation for High-Dimensional Data [2.9901605297536027]
  We introduce fastHDMI, a Python package designed for efficient variable screening in high-dimensional datasets. This work pioneers the application of three mutual information estimation methods for neuroimaging variable selection.
  arXiv  Detail & Related papers  (2024-10-14T01:49:53Z)
• Statistical Test for Auto Feature Engineering by Selective Inference [12.703556860454565]
  Auto Feature Engineering (AFE) plays a crucial role in developing practical machine learning pipelines. We propose a new statistical test for generated features by AFE algorithms based on a framework called selective inference. The proposed test can quantify the statistical significance of the generated features in the form of $p$-values, enabling theoretically guaranteed control of the risk of false findings.
  arXiv  Detail & Related papers  (2024-10-13T12:26:51Z)
• Detecting and Identifying Selection Structure in Sequential Data [53.24493902162797]
  We argue that the selective inclusion of data points based on latent objectives is common in practical situations, such as music sequences. We show that selection structure is identifiable without any parametric assumptions or interventional experiments. We also propose a provably correct algorithm to detect and identify selection structures as well as other types of dependencies.
  arXiv  Detail & Related papers  (2024-06-29T20:56:34Z)
• Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models [36.22392593103493]
  Data selection for fine-tuning large language models (LLMs) aims to choose a high-quality subset from existing datasets. Existing surveys overlook an in-depth exploration of the fine-tuning phase. We introduce a novel three-stage scheme - comprising feature extraction, criteria design, and selector evaluation - to systematically categorize and evaluate these methods.
  arXiv  Detail & Related papers  (2024-06-20T08:58:58Z)
• Globally-Optimal Greedy Experiment Selection for Active Sequential Estimation [1.1530723302736279]
  We study the problem of active sequential estimation, which involves adaptively selecting experiments for sequentially collected data. The goal is to design experiment selection rules for more accurate model estimation. We propose a class of greedy experiment selection methods and provide statistical analysis for the maximum likelihood.
  arXiv  Detail & Related papers  (2024-02-13T17:09:29Z)
• Causal Feature Selection via Transfer Entropy [59.999594949050596]
  Causal discovery aims to identify causal relationships between features with observational data. We introduce a new causal feature selection approach that relies on the forward and backward feature selection procedures. We provide theoretical guarantees on the regression and classification errors for both the exact and the finite-sample cases.
  arXiv  Detail & Related papers  (2023-10-17T08:04:45Z)
• Towards Free Data Selection with General-Purpose Models [71.92151210413374]
  A desirable data selection algorithm can efficiently choose the most informative samples to maximize the utility of limited annotation budgets. Current approaches, represented by active learning methods, typically follow a cumbersome pipeline that iterates the time-consuming model training and batch data selection repeatedly. FreeSel bypasses the heavy batch selection process, achieving a significant improvement in efficiency and being 530x faster than existing active learning methods.
  arXiv  Detail & Related papers  (2023-09-29T15:50:14Z)
• Measuring Statistical Dependencies via Maximum Norm and Characteristic Functions [0.0]
  We propose a statistical dependence measure based on the maximum-norm of the difference between joint and product-marginal characteristic functions. The proposed measure can detect arbitrary statistical dependence between two random vectors of possibly different dimensions. We conduct experiments both with simulated and real data.
  arXiv  Detail & Related papers  (2022-08-16T20:24:31Z)
• Statistical inference of travelers' route choice preferences with system-level data [4.120057972557892]
  We develop a methodology to estimate travelers' utility functions with multiple attributes using system-level data. Experiments on synthetic data show that the coefficients are consistently recovered and that hypothesis tests are a reliable statistic to identify which attributes are determinants of travelers' route choices. The methodology is also deployed at a large scale using real Fresnoworld multisource data collected during the COVID outbreak.
  arXiv  Detail & Related papers  (2022-04-23T00:38:32Z)
• Differential privacy and robust statistics in high dimensions [49.50869296871643]
  High-dimensional Propose-Test-Release (HPTR) builds upon three crucial components: the exponential mechanism, robust statistics, and the Propose-Test-Release mechanism. We show that HPTR nearly achieves the optimal sample complexity under several scenarios studied in the literature.
  arXiv  Detail & Related papers  (2021-11-12T06:36:40Z)
• Online Active Model Selection for Pre-trained Classifiers [72.84853880948894]
  We design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and streams.
  arXiv  Detail & Related papers  (2020-10-19T19:53:15Z)
• Causal Feature Selection for Algorithmic Fairness [61.767399505764736]
  We consider fairness in the integration component of data management. We propose an approach to identify a sub-collection of features that ensure the fairness of the dataset.
  arXiv  Detail & Related papers  (2020-06-10T20:20:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.