Separability and Scatteredness (S&S) Ratio-Based Efficient SVM Regularization Parameter, Kernel, and Kernel Parameter Selection

• URL: http://arxiv.org/abs/2305.10219v1
• Date: Wed, 17 May 2023 13:51:43 GMT
• Title: Separability and Scatteredness (S&S) Ratio-Based Efficient SVM Regularization Parameter, Kernel, and Kernel Parameter Selection
• Authors: Mahdi Shamsi and Soosan Beheshti
• Abstract summary: Support Vector Machine (SVM) is a robust machine learning algorithm with broad applications in classification, regression, and outlier detection. This work shows that the SVM performance can be modeled as a function of separability and scatteredness (S&S) of the data.
• Score: 10.66048003460524
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Support Vector Machine (SVM) is a robust machine learning algorithm with broad applications in classification, regression, and outlier detection. SVM requires tuning the regularization parameter (RP) which controls the model capacity and the generalization performance. Conventionally, the optimum RP is found by comparison of a range of values through the Cross-Validation (CV) procedure. In addition, for non-linearly separable data, the SVM uses kernels where a set of kernels, each with a set of parameters, denoted as a grid of kernels, are considered. The optimal choice of RP and the grid of kernels is through the grid-search of CV. By stochastically analyzing the behavior of the regularization parameter, this work shows that the SVM performance can be modeled as a function of separability and scatteredness (S&S) of the data. Separability is a measure of the distance between classes, and scatteredness is the ratio of the spread of data points. In particular, for the hinge loss cost function, an S&S ratio-based table provides the optimum RP. The S&S ratio is a powerful value that can automatically detect linear or non-linear separability before using the SVM algorithm. The provided S&S ratio-based table can also provide the optimum kernel and its parameters before using the SVM algorithm. Consequently, the computational complexity of the CV grid-search is reduced to only one time use of the SVM. The simulation results on the real dataset confirm the superiority and efficiency of the proposed approach in the sense of computational complexity over the grid-search CV method.

Related papers

• An Autotuning-based Optimization Framework for Mixed-kernel SVM Classifications in Smart Pixel Datasets and Heterojunction Transistors [0.0]
  Support Vector Machine (SVM) is a state-of-the-art classification method widely used in science and engineering. We propose an autotuning-based optimization framework to quantify the ranges of hyper parameters in SVMs to identify their optimal choices.
  arXiv  Detail & Related papers  (2024-06-26T15:50:13Z)
• Online Variational Sequential Monte Carlo [49.97673761305336]
  We build upon the variational sequential Monte Carlo (VSMC) method, which provides computationally efficient and accurate model parameter estimation and Bayesian latent-state inference. Online VSMC is capable of performing efficiently, entirely on-the-fly, both parameter estimation and particle proposal adaptation.
  arXiv  Detail & Related papers  (2023-12-19T21:45:38Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• Cost-sensitive probabilistic predictions for support vector machines [1.743685428161914]
  Support vector machines (SVMs) are widely used and constitute one of the best examined and used machine learning models. We propose a novel approach to generate probabilistic outputs for the SVM.
  arXiv  Detail & Related papers  (2023-10-09T11:00:17Z)
• Efficient Convex Algorithms for Universal Kernel Learning [50.877957471649395]
  An ideal set of kernels should: admit a linear parameterization (for tractability); dense in the set of all kernels (for accuracy) Previous algorithms for optimization of kernels were limited to classification and relied on computationally complex Semidefinite Programming (SDP) algorithms. We propose a SVD-QCQPQP algorithm which dramatically reduces the computational complexity as compared with previous SDP-based approaches.
  arXiv  Detail & Related papers  (2023-04-15T04:57:37Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• SreaMRAK a Streaming Multi-Resolution Adaptive Kernel Algorithm [60.61943386819384]
  Existing implementations of KRR require that all the data is stored in the main memory. We propose StreaMRAK - a streaming version of KRR. We present a showcase study on two synthetic problems and the prediction of the trajectory of a double pendulum.
  arXiv  Detail & Related papers  (2021-08-23T21:03:09Z)
• Estimating Average Treatment Effects with Support Vector Machines [77.34726150561087]
  Support vector machine (SVM) is one of the most popular classification algorithms in the machine learning literature. We adapt SVM as a kernel-based weighting procedure that minimizes the maximum mean discrepancy between the treatment and control groups. We characterize the bias of causal effect estimation arising from this trade-off, connecting the proposed SVM procedure to the existing kernel balancing methods.
  arXiv  Detail & Related papers  (2021-02-23T20:22:56Z)
• Radial basis function kernel optimization for Support Vector Machine classifiers [4.888981184420116]
  OKSVM is an algorithm that automatically learns the RBF kernel hyper parameter and adjusts the SVM weights simultaneously. We analyze the performance of our approach with respect to the classical SVM for classification on synthetic and real data.
  arXiv  Detail & Related papers  (2020-07-16T10:09:15Z)
• Behavioral analysis of support vector machine classifier with Gaussian kernel and imbalanced data [6.028247638616058]
  The behavior of the SVM classification model is analyzed when these parameters take different values with balanced and imbalanced data. From this analysis, we proposed a novel search algorithm. In our algorithm, we search for the optimal SVM parameters into two one-dimensional spaces instead of searching into one two-dimensional space.
  arXiv  Detail & Related papers  (2020-07-09T19:28:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.