Distributionally Robust Feature Selection

• URL: http://arxiv.org/abs/2510.21113v1
• Date: Fri, 24 Oct 2025 03:03:30 GMT
• Title: Distributionally Robust Feature Selection
• Authors: Maitreyi Swaroop, Tamar Krishnamurti, Bryan Wilder,
• Abstract summary: We study the problem of selecting limited features to observe such that models trained on them can perform well simultaneously across multiple subpopulations.<n>Our method frames the problem as a continuous relaxation of traditional variable selection using a noising mechanism.<n>We develop a model-agnostic framework that balances overall performance of downstream prediction across populations.
• Score: 14.493253907785473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the problem of selecting limited features to observe such that models trained on them can perform well simultaneously across multiple subpopulations. This problem has applications in settings where collecting each feature is costly, e.g. requiring adding survey questions or physical sensors, and we must be able to use the selected features to create high-quality downstream models for different populations. Our method frames the problem as a continuous relaxation of traditional variable selection using a noising mechanism, without requiring backpropagation through model training processes. By optimizing over the variance of a Bayes-optimal predictor, we develop a model-agnostic framework that balances overall performance of downstream prediction across populations. We validate our approach through experiments on both synthetic datasets and real-world data.

Related papers

• Heterogeneous Self-Supervised Acoustic Pre-Training with Local Constraints [64.15709757611369]
  We propose a new self-supervised pre-training approach to dealing with heterogeneous data.<n>The proposed approach can significantly improve the adaptivity of the self-supervised pre-trained model for the downstream supervised fine-tuning tasks.
  arXiv  Detail & Related papers  (2025-08-27T15:48:50Z)
• Diffusion-Driven High-Dimensional Variable Selection [6.993247097440294]
  We propose a resample-aggregate framework that exploits diffusion models' ability to generate high-fidelity synthetic data.<n>We show that the proposed method is selection consistent under mild assumptions.<n>Our method advances variable selection methodology and broadens the toolkit for interpretable, statistically rigorous analysis.
  arXiv  Detail & Related papers  (2025-08-19T14:54:20Z)
• Diversified Batch Selection for Training Acceleration [68.67164304377732]
  A prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner. We propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples.
  arXiv  Detail & Related papers  (2024-06-07T12:12:20Z)
• Diffusion Models as Constrained Samplers for Optimization with Unknown Constraints [55.39203337683045]
  We propose to perform optimization within the data manifold using diffusion models.<n>Depending on the differentiability of the objective function, we propose two different sampling methods.<n>Our method achieves better or comparable performance with previous state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-28T03:09:12Z)
• Model Assessment and Selection under Temporal Distribution Shift [1.024113475677323]
  We develop an adaptive rolling window approach to estimate the generalization error of a given model. We also integrate pairwise comparisons into a single-elimination tournament, achieving near-optimal model selection from a collection of candidates.
  arXiv  Detail & Related papers  (2024-02-13T18:54:08Z)
• Bilevel Optimization for Feature Selection in the Data-Driven Newsvendor Problem [8.281391209717105]
  We study the feature-based news vendor problem, in which a decision-maker has access to historical data. In this setting, we investigate feature selection, aiming to derive sparse, explainable models with improved out-of-sample performance. We present a mixed integer linear program reformulation for the bilevel program, which can be solved to optimality with standard optimization solvers.
  arXiv  Detail & Related papers  (2022-09-12T08:52:26Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Model Selection for Bayesian Autoencoders [25.619565817793422]
  We propose to optimize the distributional sliced-Wasserstein distance between the output of the autoencoder and the empirical data distribution. We turn our BAE into a generative model by fitting a flexible Dirichlet mixture model in the latent space. We evaluate our approach qualitatively and quantitatively using a vast experimental campaign on a number of unsupervised learning tasks and show that, in small-data regimes where priors matter, our approach provides state-of-the-art results.
  arXiv  Detail & Related papers  (2021-06-11T08:55:00Z)
• A Twin Neural Model for Uplift [59.38563723706796]
  Uplift is a particular case of conditional treatment effect modeling. We propose a new loss function defined by leveraging a connection with the Bayesian interpretation of the relative risk. We show our proposed method is competitive with the state-of-the-art in simulation setting and on real data from large scale randomized experiments.
  arXiv  Detail & Related papers  (2021-05-11T16:02:39Z)
• Characterizing Fairness Over the Set of Good Models Under Selective Labels [69.64662540443162]
  We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance. We provide tractable algorithms to compute the range of attainable group-level predictive disparities. We extend our framework to address the empirically relevant challenge of selectively labelled data.
  arXiv  Detail & Related papers  (2021-01-02T02:11:37Z)
• Reinforced Data Sampling for Model Diversification [15.547681142342846]
  This paper proposes a new Reinforced Data Sampling (RDS) method to learn how to sample data adequately. We formulate the optimisation problem of model diversification $delta-div$ in data sampling to maximise learning potentials and optimum allocation by injecting model diversity. Our results suggest that the trainable sampling for model diversification is useful for competition organisers, researchers, or even starters to pursue full potentials of various machine learning tasks.
  arXiv  Detail & Related papers  (2020-06-12T11:46:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.