Optimal Model Averaging of Support Vector Machines in Diverging Model Spaces

• URL: http://arxiv.org/abs/2112.12961v1
• Date: Fri, 24 Dec 2021 06:31:51 GMT
• Title: Optimal Model Averaging of Support Vector Machines in Diverging Model Spaces
• Authors: Chaoxia Yuan, Chao Ying, Zhou Yu, Fang Fang
• Abstract summary: Support vector machine (SVM) is a powerful classification method that has achieved great success in many fields. We propose a frequentist model averaging procedure for SVM which selects the optimal weight by cross validation. We show optimality of the proposed method in the sense that the ratio of its hinge loss to the lowest possible loss converges to one.
• Score: 34.0132394492309
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Support vector machine (SVM) is a powerful classification method that has achieved great success in many fields. Since its performance can be seriously impaired by redundant covariates, model selection techniques are widely used for SVM with high dimensional covariates. As an alternative to model selection, significant progress has been made in the area of model averaging in the past decades. Yet no frequentist model averaging method was considered for SVM. This work aims to fill the gap and to propose a frequentist model averaging procedure for SVM which selects the optimal weight by cross validation. Even when the number of covariates diverges at an exponential rate of the sample size, we show asymptotic optimality of the proposed method in the sense that the ratio of its hinge loss to the lowest possible loss converges to one. We also derive the convergence rate which provides more insights to model averaging. Compared to model selection methods of SVM which require a tedious but critical task of tuning parameter selection, the model averaging method avoids the task and shows promising performances in the empirical studies.

Related papers

• Computation-Aware Gaussian Processes: Model Selection And Linear-Time Inference [55.150117654242706]
  We show that model selection for computation-aware GPs trained on 1.8 million data points can be done within a few hours on a single GPU. As a result of this work, Gaussian processes can be trained on large-scale datasets without significantly compromising their ability to quantify uncertainty.
  arXiv  Detail & Related papers  (2024-11-01T21:11:48Z)
• Annealed Stein Variational Gradient Descent for Improved Uncertainty Estimation in Full-Waveform Inversion [25.714206592953545]
  Variational Inference (VI) provides an approximate solution to the posterior distribution in the form of a parametric or non-parametric proposal distribution. This study aims to improve the performance of VI within the context of Full-Waveform Inversion.
  arXiv  Detail & Related papers  (2024-10-17T06:15:26Z)
• On the Laplace Approximation as Model Selection Criterion for Gaussian Processes [6.990493129893112]
  We introduce multiple metrics based on the Laplace approximation. Experiments show that our metrics are comparable in quality to the gold standard dynamic nested sampling.
  arXiv  Detail & Related papers  (2024-03-14T09:28:28Z)
• Robust Twin Parametric Margin Support Vector Machine for Multiclass Classification [0.0]
  We present novel Twin Parametric Margin Support Vector Machine (TPMSVM) models to tackle the problem of multiclass classification. We construct bounded-by-norm uncertainty sets around each sample and derive the robust counterpart of deterministic models. We test the proposed TPMSVM methodology on real-world datasets, showing the good performance of the approach.
  arXiv  Detail & Related papers  (2023-06-09T19:27:24Z)
• Numerical Optimizations for Weighted Low-rank Estimation on Language Model [73.12941276331316]
  Singular value decomposition (SVD) is one of the most popular compression methods that approximates a target matrix with smaller matrices. Standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. We show that our method can perform better than current SOTA methods in neural-based language models.
  arXiv  Detail & Related papers  (2022-11-02T00:58:02Z)
• RoMA: Robust Model Adaptation for Offline Model-based Optimization [115.02677045518692]
  We consider the problem of searching an input maximizing a black-box objective function given a static dataset of input-output queries. A popular approach to solving this problem is maintaining a proxy model that approximates the true objective function. Here, the main challenge is how to avoid adversarially optimized inputs during the search.
  arXiv  Detail & Related papers  (2021-10-27T05:37:12Z)
• A Data-driven feature selection and machine-learning model benchmark for the prediction of longitudinal dispersion coefficient [29.58577229101903]
  An accurate prediction on Longitudinal Dispersion(LD) coefficient can produce a performance leap in related simulation. In this study, a global optimal feature set was proposed through numerical comparison of the distilled local optimums in performance with representative ML models. Results show that the support vector machine has significantly better performance than other models.
  arXiv  Detail & Related papers  (2021-07-16T09:50:38Z)
• Robust Regression via Model Based Methods [13.300549123177705]
  We propose an algorithm inspired by so-called model-based optimization (MBO) [35, 36], which replaces a non-objective with a convex model function. We apply this to robust regression, proposing SADM, a function of the Online Alternating Direction Method of Multipliers (OOADM) [50] to solve the inner optimization in MBO. Finally, we demonstrate experimentally (a) the robustness of l_p norms to outliers and (b) the efficiency of our proposed model-based algorithms in comparison with methods on autoencoders and multi-target regression.
  arXiv  Detail & Related papers  (2021-06-20T21:45:35Z)
• Estimating Average Treatment Effects with Support Vector Machines [77.34726150561087]
  Support vector machine (SVM) is one of the most popular classification algorithms in the machine learning literature. We adapt SVM as a kernel-based weighting procedure that minimizes the maximum mean discrepancy between the treatment and control groups. We characterize the bias of causal effect estimation arising from this trade-off, connecting the proposed SVM procedure to the existing kernel balancing methods.
  arXiv  Detail & Related papers  (2021-02-23T20:22:56Z)
• Robust Finite Mixture Regression for Heterogeneous Targets [70.19798470463378]
  We propose an FMR model that finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously. We provide non-asymptotic oracle performance bounds for our model under a high-dimensional learning framework. The results show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-12T03:27:07Z)
• Robust, Accurate Stochastic Optimization for Variational Inference [68.83746081733464]
  We show that common optimization methods lead to poor variational approximations if the problem is moderately large. Motivated by these findings, we develop a more robust and accurate optimization framework by viewing the underlying algorithm as producing a Markov chain.
  arXiv  Detail & Related papers  (2020-09-01T19:12:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.