Factorization Machines with Regularization for Sparse Feature Interactions

• URL: http://arxiv.org/abs/2010.09225v2
• Date: Thu, 1 Apr 2021 03:51:12 GMT
• Title: Factorization Machines with Regularization for Sparse Feature Interactions
• Authors: Kyohei Atarashi, Satoshi Oyama, Masahito Kurihara
• Abstract summary: Factorization machines (FMs) are machine learning predictive models based on second-order feature interactions. We present a new regularization scheme for feature interaction selection in FMs. For feature interaction selection, our proposed regularizer makes the feature interaction matrix sparse without a restriction on sparsity patterns imposed by the existing methods.
• Score: 13.593781209611112
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Factorization machines (FMs) are machine learning predictive models based on second-order feature interactions and FMs with sparse regularization are called sparse FMs. Such regularizations enable feature selection, which selects the most relevant features for accurate prediction, and therefore they can contribute to the improvement of the model accuracy and interpretability. However, because FMs use second-order feature interactions, the selection of features often causes the loss of many relevant feature interactions in the resultant models. In such cases, FMs with regularization specially designed for feature interaction selection trying to achieve interaction-level sparsity may be preferred instead of those just for feature selection trying to achieve feature-level sparsity. In this paper, we present a new regularization scheme for feature interaction selection in FMs. The proposed regularizer is an upper bound of the $\ell_1$ regularizer for the feature interaction matrix, which is computed from the parameter matrix of FMs. For feature interaction selection, our proposed regularizer makes the feature interaction matrix sparse without a restriction on sparsity patterns imposed by the existing methods. We also describe efficient proximal algorithms for the proposed FMs and present theoretical analyses of both existing and the new regularize. In addition, we will discuss how our ideas can be applied or extended to more accurate feature selection and other related models such as higher-order FMs and the all-subsets model. The analysis and experimental results on synthetic and real-world datasets show the effectiveness of the proposed methods.

Related papers

• Causal Feature Selection via Transfer Entropy [59.999594949050596]
  Causal discovery aims to identify causal relationships between features with observational data. We introduce a new causal feature selection approach that relies on the forward and backward feature selection procedures. We provide theoretical guarantees on the regression and classification errors for both the exact and the finite-sample cases.
  arXiv  Detail & Related papers  (2023-10-17T08:04:45Z)
• Opening the random forest black box by the analysis of the mutual impact of features [0.0]
  We propose two novel approaches that focus on the mutual impact of features in random forests. MFI and MIR are very promising to shed light on the complex relationships between features and outcome.
  arXiv  Detail & Related papers  (2023-04-05T15:03:46Z)
• Meta-Wrapper: Differentiable Wrapping Operator for User Interest Selection in CTR Prediction [97.99938802797377]
  Click-through rate (CTR) prediction, whose goal is to predict the probability of the user to click on an item, has become increasingly significant in recommender systems. Recent deep learning models with the ability to automatically extract the user interest from his/her behaviors have achieved great success. We propose a novel approach under the framework of the wrapper method, which is named Meta-Wrapper.
  arXiv  Detail & Related papers  (2022-06-28T03:28:15Z)
• Boosting Factorization Machines via Saliency-Guided Mixup [125.15872106335692]
  We present MixFM, inspired by Mixup, to generate auxiliary training data to boost Factorization machines (FMs) We also put forward a novel Factorization Machine powered by Saliency-guided Mixup (denoted as SMFM)
  arXiv  Detail & Related papers  (2022-06-17T09:49:00Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Understanding Interlocking Dynamics of Cooperative Rationalization [90.6863969334526]
  Selective rationalization explains the prediction of complex neural networks by finding a small subset of the input that is sufficient to predict the neural model output. We reveal a major problem with such cooperative rationalization paradigm -- model interlocking. We propose a new rationalization framework, called A2R, which introduces a third component into the architecture, a predictor driven by soft attention as opposed to selection.
  arXiv  Detail & Related papers  (2021-10-26T17:39:18Z)
• Memory-Efficient Factorization Machines via Binarizing both Data and Model Coefficients [9.692334398809457]
  Subspace imating machine (SEFM) has been proposed to overcome the limitation of Factorization Machines (FM) We propose a new method called Binarized FM which constraints the model parameters to be binary values. Our proposed method achieves comparable accuracy with SEFM but with much less memory cost.
  arXiv  Detail & Related papers  (2021-08-17T03:30:52Z)
• Top-$k$ Regularization for Supervised Feature Selection [11.927046591097623]
  We introduce a novel, simple yet effective regularization approach, named top-$k$ regularization, to supervised feature selection. We show that the top-$k$ regularization is effective and stable for supervised feature selection.
  arXiv  Detail & Related papers  (2021-06-04T01:12:47Z)
• GraphFM: Graph Factorization Machines for Feature Interaction Modeling [27.307086868266012]
  We propose a novel approach, Graph Factorization Machine (GraphFM), by naturally representing features in the graph structure. In particular, we design a mechanism to select the beneficial feature interactions and formulate them as edges between features. The proposed model integrates the interaction function of FM into the feature aggregation strategy of Graph Neural Network (GNN)
  arXiv  Detail & Related papers  (2021-05-25T12:10:54Z)
• Quaternion Factorization Machines: A Lightweight Solution to Intricate Feature Interaction Modelling [76.89779231460193]
  factorization machine (FM) is capable of automatically learning high-order interactions among features to make predictions without the need for manual feature engineering. We propose the quaternion factorization machine (QFM) and quaternion neural factorization machine (QNFM) for sparse predictive analytics.
  arXiv  Detail & Related papers  (2021-04-05T00:02:36Z)
• Information-theoretic Feature Selection via Tensor Decomposition and Submodularity [38.05393186002834]
  We introduce a low-rank tensor model of the joint PMF of all variables and indirect targeting as a way of mitigating complexity and maximizing the classification performance for a given number of features. By indirectly aiming to predict the latent variable of the naive Bayes model instead of the original target variable, it is possible to formulate the feature selection problem as of a monotone submodular function subject to a cardinality constraint.
  arXiv  Detail & Related papers  (2020-10-30T10:36:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.