MINERVA: Mutual Information Neural Estimation for Supervised Feature Selection

• URL: http://arxiv.org/abs/2510.02610v2
• Date: Mon, 06 Oct 2025 09:40:13 GMT
• Title: MINERVA: Mutual Information Neural Estimation for Supervised Feature Selection
• Authors: Taurai Muvunza, Egor Kraev, Pere Planell-Morell, Alexander Y. Shestopaloff,
• Abstract summary: We introduce a novel approach to supervised feature selection based on neural estimation of mutual information between features and targets.<n>We paramaterize the approximation of mutual information with neural networks and perform feature selection using a carefully designed loss function augmented with sparsity-inducing regularizers.<n>Our method is implemented in a two-stage process to decouple representation learning from feature selection, ensuring better generalization and a more accurate expression of feature importance.
• Score: 39.57737590420284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing feature filters rely on statistical pair-wise dependence metrics to model feature-target relationships, but this approach may fail when the target depends on higher-order feature interactions rather than individual contributions. We introduce Mutual Information Neural Estimation Regularized Vetting Algorithm (MINERVA), a novel approach to supervised feature selection based on neural estimation of mutual information between features and targets. We paramaterize the approximation of mutual information with neural networks and perform feature selection using a carefully designed loss function augmented with sparsity-inducing regularizers. Our method is implemented in a two-stage process to decouple representation learning from feature selection, ensuring better generalization and a more accurate expression of feature importance. We present examples of ubiquitous dependency structures that are rarely captured in literature and show that our proposed method effectively captures these complex feature-target relationships by evaluating feature subsets as an ensemble. Experimental results on synthetic and real-life fraud datasets demonstrate the efficacy of our method and its ability to perform exact solutions.

Related papers

• Detecting and Pruning Prominent but Detrimental Neurons in Large Language Models [68.57424628540907]
  Large language models (LLMs) often develop learned mechanisms specialized to specific datasets.<n>We introduce a fine-tuning approach designed to enhance generalization by identifying and pruning neurons associated with dataset-specific mechanisms.<n>Our method employs Integrated Gradients to quantify each neuron's influence on high-confidence predictions, pinpointing those that disproportionately contribute to dataset-specific performance.
  arXiv  Detail & Related papers  (2025-07-12T08:10:10Z)
• IGANN Sparse: Bridging Sparsity and Interpretability with Non-linear Insight [4.010646933005848]
  IGANN Sparse is a novel machine learning model from the family of generalized additive models. It promotes sparsity through a non-linear feature selection process during training. This ensures interpretability through improved model sparsity without sacrificing predictive performance.
  arXiv  Detail & Related papers  (2024-03-17T22:44:36Z)
• Causal Feature Selection via Transfer Entropy [59.999594949050596]
  Causal discovery aims to identify causal relationships between features with observational data. We introduce a new causal feature selection approach that relies on the forward and backward feature selection procedures. We provide theoretical guarantees on the regression and classification errors for both the exact and the finite-sample cases.
  arXiv  Detail & Related papers  (2023-10-17T08:04:45Z)
• Copula for Instance-wise Feature Selection and Ranking [24.09326839818306]
  We propose to incorporate the Gaussian copula, a powerful mathematical technique for capturing correlations between variables, into the current feature selection framework. Experimental results on both synthetic and real datasets, in terms of performance comparison and interpretability, demonstrate that our method is capable of capturing meaningful correlations.
  arXiv  Detail & Related papers  (2023-08-01T13:45:04Z)
• Regularization Through Simultaneous Learning: A Case Study on Plant Classification [0.0]
  This paper introduces Simultaneous Learning, a regularization approach drawing on principles of Transfer Learning and Multi-task Learning. We leverage auxiliary datasets with the target dataset, the UFOP-HVD, to facilitate simultaneous classification guided by a customized loss function. Remarkably, our approach demonstrates superior performance over models without regularization.
  arXiv  Detail & Related papers  (2023-05-22T19:44:57Z)
• A Two-Stage Feature Selection Approach for Robust Evaluation of Treatment Effects in High-Dimensional Observational Data [1.4710887888397084]
  We propose a novel two-stage feature selection technique called, Outcome Adaptive Elastic Net (OAENet) OAENet is explicitly designed for making robust causal inference decisions using matching techniques. Numerical experiments on simulated data demonstrate that OAENet significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-27T02:54:30Z)
• Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization [77.8735802150511]
  We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-11T21:17:02Z)
• Detecting Beneficial Feature Interactions for Recommender Systems [15.599904548629537]
  Feature interactions are essential for achieving high accuracy in recommender systems. We propose a graph neural network approach to effectively model them, together with a novel technique to automatically detect those feature interactions. Our proposed model is proved to be effective through the information bottleneck principle and statistical interaction theory.
  arXiv  Detail & Related papers  (2020-08-02T06:08:23Z)
• Out-of-distribution Generalization via Partial Feature Decorrelation [72.96261704851683]
  We present a novel Partial Feature Decorrelation Learning (PFDL) algorithm, which jointly optimize a feature decomposition network and the target image classification model. The experiments on real-world datasets demonstrate that our method can improve the backbone model's accuracy on OOD image classification datasets.
  arXiv  Detail & Related papers  (2020-07-30T05:48:48Z)
• A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention [96.77554122595578]
  We introduce a parametrized representation of fixed size, which embeds and then aggregates elements from a given input set according to the optimal transport plan between the set and a trainable reference. Our approach scales to large datasets and allows end-to-end training of the reference, while also providing a simple unsupervised learning mechanism with small computational cost.
  arXiv  Detail & Related papers  (2020-06-22T08:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.