Binary Feature Mask Optimization for Feature Selection

• URL: http://arxiv.org/abs/2401.12644v1
• Date: Tue, 23 Jan 2024 10:54:13 GMT
• Title: Binary Feature Mask Optimization for Feature Selection
• Authors: Mehmet E. Lorasdagi, Mehmet Y. Turali, Ali T. Koc, Suleyman S. Kozat
• Abstract summary: We introduce a novel framework that selects features considering the predictions of the model. Our framework innovates by using a novel feature masking approach to eliminate the features during the selection process. We demonstrate significant performance improvements on the real-life datasets using LightGBM and Multi-Layer Perceptron as our ML models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We investigate feature selection problem for generic machine learning (ML) models. We introduce a novel framework that selects features considering the predictions of the model. Our framework innovates by using a novel feature masking approach to eliminate the features during the selection process, instead of completely removing them from the dataset. This allows us to use the same ML model during feature selection, unlike other feature selection methods where we need to train the ML model again as the dataset has different dimensions on each iteration. We obtain the mask operator using the predictions of the ML model, which offers a comprehensive view on the subsets of the features essential for the predictive performance of the model. A variety of approaches exist in the feature selection literature. However, no study has introduced a training-free framework for a generic ML model to select features while considering the importance of the feature subsets as a whole, instead of focusing on the individual features. We demonstrate significant performance improvements on the real-life datasets under different settings using LightGBM and Multi-Layer Perceptron as our ML models. Additionally, we openly share the implementation code for our methods to encourage the research and the contributions in this area.

Related papers

• Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild [84.57103623507082]
  This paper introduces Model-GLUE, a holistic Large Language Models scaling guideline. Our work starts with a benchmarking of existing LLM scaling techniques, especially selective merging, and variants of mixture. Our methodology involves the clustering of mergeable models and optimal merging strategy selection, and the integration of clusters through a model mixture.
  arXiv  Detail & Related papers  (2024-10-07T15:55:55Z)
• Rethinking Pre-trained Feature Extractor Selection in Multiple Instance Learning for Whole Slide Image Classification [2.6703221234079946]
  Multiple instance learning (MIL) has become a preferred method for gigapixel whole slide image (WSI) classification without requiring patch-level annotations. This study systematically evaluating MIL feature extractors across three dimensions: pre-training dataset, backbone model, and pre-training method. Our findings reveal that selecting a robust self-supervised learning (SSL) method has a greater impact on performance than relying solely on an in-domain pre-training dataset.
  arXiv  Detail & Related papers  (2024-08-02T10:34:23Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets [106.7760874400261]
  This paper presents ML-SUPERB2.0, which is a new benchmark for evaluating pre-trained SSL and supervised speech models. We find performance improvements over the setup of ML-SUPERB, but performance depends on the downstream model design. Also, we find large performance differences between languages and datasets, suggesting the need for more targeted approaches.
  arXiv  Detail & Related papers  (2024-06-12T21:01:26Z)
• REFRESH: Responsible and Efficient Feature Reselection Guided by SHAP Values [17.489279048199304]
  REFRESH is a method to reselect features so that additional constraints that are desirable towards model performance can be achieved without having to train several new models. REFRESH's underlying algorithm is a novel technique using SHAP values and correlation analysis that can approximate for the predictions of a model without having to train these models.
  arXiv  Detail & Related papers  (2024-03-13T18:06:43Z)
• Embedded feature selection in LSTM networks with multi-objective evolutionary ensemble learning for time series forecasting [49.1574468325115]
  We present a novel feature selection method embedded in Long Short-Term Memory networks. Our approach optimize the weights and biases of the LSTM in a partitioned manner. Experimental evaluations on air quality time series data from Italy and southeast Spain demonstrate that our method substantially improves the ability generalization of conventional LSTMs.
  arXiv  Detail & Related papers  (2023-12-29T08:42:10Z)
• Extending Variability-Aware Model Selection with Bias Detection in Machine Learning Projects [0.7646713951724013]
  This paper describes work on extending an adaptive variability-aware model selection method with bias detection in machine learning projects. The proposed approach aims to advance the state of the art by making explicit factors that influence model selection, particularly those related to bias, as well as their interactions.
  arXiv  Detail & Related papers  (2023-11-23T22:08:29Z)
• Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning [47.651130958272155]
  Underspecification refers to the existence of multiple models that are indistinguishable in their in-domain accuracy. We formalize the concept of underspecification and propose a method to identify and partially address it.
  arXiv  Detail & Related papers  (2022-07-06T11:20:40Z)
• Filter Methods for Feature Selection in Supervised Machine Learning Applications -- Review and Benchmark [0.0]
  This review synthesizes the literature on feature selection benchmarking and evaluates the performance of 58 methods in the widely used R environment. We consider four typical dataset scenarios that are challenging for ML models.
  arXiv  Detail & Related papers  (2021-11-23T20:20:24Z)
• Compositional Fine-Grained Low-Shot Learning [58.53111180904687]
  We develop a novel compositional generative model for zero- and few-shot learning to recognize fine-grained classes with a few or no training samples. We propose a feature composition framework that learns to extract attribute features from training samples and combines them to construct fine-grained features for rare and unseen classes.
  arXiv  Detail & Related papers  (2021-05-21T16:18:24Z)
• Feature Selection Methods for Uplift Modeling and Heterogeneous Treatment Effect [1.349645012479288]
  Uplift modeling is a causal learning technique that estimates subgroup-level treatment effects. Traditional methods for doing feature selection are not fit for the task. We introduce a set of feature selection methods explicitly designed for uplift modeling.
  arXiv  Detail & Related papers  (2020-05-05T00:28:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.