More Powerful and General Selective Inference for Stepwise Feature Selection using the Homotopy Continuation Approach

• URL: http://arxiv.org/abs/2012.13545v2
• Date: Thu, 22 Apr 2021 01:59:45 GMT
• Title: More Powerful and General Selective Inference for Stepwise Feature Selection using the Homotopy Continuation Approach
• Authors: Kazuya Sugiyama, Vo Nguyen Le Duy, Ichiro Takeuchi
• Abstract summary: Conditional selective inference (SI) has been actively studied as a new statistical inference framework for data-driven hypotheses. The main limitation of the existing conditional SI methods is the loss of power due to over-conditioning. We develop a more powerful and general conditional SI method for SFS using the homotopy method which enables us to overcome this limitation.
• Score: 17.191349670354228
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conditional selective inference (SI) has been actively studied as a new statistical inference framework for data-driven hypotheses. The basic idea of conditional SI is to make inferences conditional on the selection event characterized by a set of linear and/or quadratic inequalities. Conditional SI has been mainly studied in the context of feature selection such as stepwise feature selection (SFS). The main limitation of the existing conditional SI methods is the loss of power due to over-conditioning, which is required for computational tractability. In this study, we develop a more powerful and general conditional SI method for SFS using the homotopy method which enables us to overcome this limitation. The homotopy-based SI is especially effective for more complicated feature selection algorithms. As an example, we develop a conditional SI method for forward-backward SFS with AIC-based stopping criteria and show that it is not adversely affected by the increased complexity of the algorithm. We conduct several experiments to demonstrate the effectiveness and efficiency of the proposed method.

Related papers

• Conditionally valid Probabilistic Conformal Prediction [57.80927226809277]
  We develop a new method for creating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. We demonstrate the effectiveness of our approach through extensive simulations, showing that it outperforms existing methods in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• Information-Theoretic State Variable Selection for Reinforcement Learning [4.2050490361120465]
  We introduce the Transfer Entropy Redundancy Criterion (TERC), an information-theoretic criterion. TERC determines if there is textitentropy transferred from state variables to actions during training. We define an algorithm based on TERC that provably excludes variables from the state that have no effect on the final performance of the agent.
  arXiv  Detail & Related papers  (2024-01-21T14:51:09Z)
• Bounded P-values in Parametric Programming-based Selective Inference [23.35466397627952]
  We introduce a procedure to reduce the computational cost while guaranteeing the desired precision, by proposing a method to compute the lower and upper bounds of p-values. We demonstrate the effectiveness of the proposed method in hypothesis testing problems for feature selection in linear models and attention region identification in deep neural networks.
  arXiv  Detail & Related papers  (2023-07-21T04:55:03Z)
• Bounding Counterfactuals under Selection Bias [60.55840896782637]
  We propose a first algorithm to address both identifiable and unidentifiable queries. We prove that, in spite of the missingness induced by the selection bias, the likelihood of the available data is unimodal.
  arXiv  Detail & Related papers  (2022-07-26T10:33:10Z)
• Interactively Learning Preference Constraints in Linear Bandits [100.78514640066565]
  We study sequential decision-making with known rewards and unknown constraints. As an application, we consider learning constraints to represent human preferences in a driving simulation.
  arXiv  Detail & Related papers  (2022-06-10T17:52:58Z)
• Fast Feature Selection with Fairness Constraints [49.142308856826396]
  We study the fundamental problem of selecting optimal features for model construction. This problem is computationally challenging on large datasets, even with the use of greedy algorithm variants. We extend the adaptive query model, recently proposed for the greedy forward selection for submodular functions, to the faster paradigm of Orthogonal Matching Pursuit for non-submodular functions. The proposed algorithm achieves exponentially fast parallel run time in the adaptive query model, scaling much better than prior work.
  arXiv  Detail & Related papers  (2022-02-28T12:26:47Z)
• Recursive Causal Structure Learning in the Presence of Latent Variables and Selection Bias [27.06618125828978]
  We consider the problem of learning the causal MAG of a system from observational data in the presence of latent variables and selection bias. We propose a novel computationally efficient constraint-based method that is sound and complete. We provide experimental results to compare the proposed approach with the state of the art on both synthetic and real-world structures.
  arXiv  Detail & Related papers  (2021-10-22T19:49:59Z)
• More Powerful Conditional Selective Inference for Generalized Lasso by Parametric Programming [20.309302270008146]
  Conditional selective inference (SI) has been studied intensively as a new statistical inference framework for data-driven hypotheses. We propose a more powerful and general conditional SI method for a class of problems that can be converted into quadratic parametric programming.
  arXiv  Detail & Related papers  (2021-05-11T10:12:00Z)
• Sinkhorn Natural Gradient for Generative Models [125.89871274202439]
  We propose a novel Sinkhorn Natural Gradient (SiNG) algorithm which acts as a steepest descent method on the probability space endowed with the Sinkhorn divergence. We show that the Sinkhorn information matrix (SIM), a key component of SiNG, has an explicit expression and can be evaluated accurately in complexity that scales logarithmically. In our experiments, we quantitatively compare SiNG with state-of-the-art SGD-type solvers on generative tasks to demonstrate its efficiency and efficacy of our method.
  arXiv  Detail & Related papers  (2020-11-09T02:51:17Z)
• Causal Feature Selection for Algorithmic Fairness [61.767399505764736]
  We consider fairness in the integration component of data management. We propose an approach to identify a sub-collection of features that ensure the fairness of the dataset.
  arXiv  Detail & Related papers  (2020-06-10T20:20:10Z)
• Parametric Programming Approach for More Powerful and General Lasso Selective Inference [25.02674598600182]
  Selective Inference (SI) has been actively studied in the past few years for conducting inference on the features of linear models. The main limitation of the original SI approach for Lasso is that the inference is conducted not only conditional on the selected features but also on their signs. We propose a parametric programming-based method that can conduct SI without conditioning on signs even when we have thousands of active features.
  arXiv  Detail & Related papers  (2020-04-21T04:46:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.