Adaptive sparseness for correntropy-based robust regression via automatic relevance determination

• URL: http://arxiv.org/abs/2302.00082v1
• Date: Tue, 31 Jan 2023 20:23:32 GMT
• Title: Adaptive sparseness for correntropy-based robust regression via automatic relevance determination
• Authors: Yuanhao Li, Badong Chen, Okito Yamashita, Natsue Yoshimura, Yasuharu Koike
• Abstract summary: We integrate the maximum correntropy criterion (MCC) based robust regression algorithm with the automatic relevance determination (ARD) technique in a Bayesian framework. We use an inherent noise assumption from the MCC to derive an explicit likelihood function, and realize the maximum a posteriori (MAP) estimation with the ARD prior. MCC-ARD achieves superior prediction performance and feature selection capability than L1-regularized MCC, as demonstrated by a noisy and high-dimensional simulation study.
• Score: 17.933460891374498
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparseness and robustness are two important properties for many machine learning scenarios. In the present study, regarding the maximum correntropy criterion (MCC) based robust regression algorithm, we investigate to integrate the MCC method with the automatic relevance determination (ARD) technique in a Bayesian framework, so that MCC-based robust regression could be implemented with adaptive sparseness. To be specific, we use an inherent noise assumption from the MCC to derive an explicit likelihood function, and realize the maximum a posteriori (MAP) estimation with the ARD prior by variational Bayesian inference. Compared to the existing robust and sparse L1-regularized MCC regression, the proposed MCC-ARD regression can eradicate the troublesome tuning for the regularization hyper-parameter which controls the regularization strength. Further, MCC-ARD achieves superior prediction performance and feature selection capability than L1-regularized MCC, as demonstrated by a noisy and high-dimensional simulation study.

Related papers

• Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models [40.216463162163976]
  We develop a batch minorization-maximization algorithm for softmax-gated multinomial-logistic MoE.<n>We also prove finite-sample rates for conditional density estimation and parameter recovery.<n>Experiments on biological protein--protein interaction prediction validate the full pipeline.
  arXiv  Detail & Related papers  (2026-02-08T14:45:41Z)
• On the Plasticity and Stability for Post-Training Large Language Models [54.757672540381236]
  We identify a root cause as the conflict between plasticity and stability gradients.<n>We propose Probabilistic Conflict Resolution (PCR), a framework that models gradients as random variables.<n>PCR significantly smooths the training trajectory and achieves superior performance in various reasoning tasks.
  arXiv  Detail & Related papers  (2026-02-06T07:31:26Z)
• Optimization and Regularization Under Arbitrary Objectives [0.0]
  This study investigates the limitations of applying Markov Chain Monte Carlo (MCMC) methods to arbitrary objective functions.<n>We show that their performance depends on the sharpness of the employed likelihood form.<n>The study concludes with a separate analysis examining the implications of extreme likelihood sharpness on arbitrary objective functions.
  arXiv  Detail & Related papers  (2025-11-24T19:03:43Z)
• Robust Iterative Learning Hidden Quantum Markov Models [0.7493761475572844]
  Hidden Quantum Markov Models (HQMMs) extend classical Hidden Markov Models to the quantum domain.<n>Existing HQMM learning algorithms are sensitive to data corruption and lack mechanisms to ensure robustness under adversarial perturbations.<n>We introduce the Adversarially Corrupted HQMM, which formalizes robustness analysis by allowing a controlled fraction of observation sequences to be adversarially corrupted.
  arXiv  Detail & Related papers  (2025-10-27T11:48:44Z)
• Bi-level Meta-Policy Control for Dynamic Uncertainty Calibration in Evidential Deep Learning [11.953394478206581]
  We propose the Meta-Policy Controller (MPC), a dynamic meta-learning framework that adjusts the KL divergence coefficient and Dirichlet prior strengths for optimal uncertainty modeling.<n>MPC significantly enhances the reliability and calibration of model predictions across various tasks, improving uncertainty calibration, prediction accuracy, and performance retention after confidence-based sample rejection.
  arXiv  Detail & Related papers  (2025-10-10T02:39:26Z)
• Confidence Optimization for Probabilistic Encoding [0.9999629695552196]
  We introduce a confidence-aware mechanism to adjust distance calculations.<n>We replace the conventional KL divergence-based variance regularization with a simpler L2 regularization term to directly constrain variance.<n>Our method significantly improves performance and generalization on both the BERT and the RoBERTa model.
  arXiv  Detail & Related papers  (2025-07-22T15:32:27Z)
• A robust and adaptive MPC formulation for Gaussian process models [2.6661512675766037]
  We present a robust and adaptive model predictive control (MPC) framework for uncertain nonlinear systems.<n>We derive robust predictions for GP models using metrics, which are incorporated in the MPC formulation.<n>We provide a numerical example of a planar quadrotor subject to difficult-to-model ground effects.
  arXiv  Detail & Related papers  (2025-07-02T19:27:14Z)
• Theoretical Insights in Model Inversion Robustness and Conditional Entropy Maximization for Collaborative Inference Systems [89.35169042718739]
  collaborative inference enables end users to leverage powerful deep learning models without exposure of sensitive raw data to cloud servers. Recent studies have revealed that these intermediate features may not sufficiently preserve privacy, as information can be leaked and raw data can be reconstructed via model inversion attacks (MIAs) This work first theoretically proves that the conditional entropy of inputs given intermediate features provides a guaranteed lower bound on the reconstruction mean square error (MSE) under any MIA. Then, we derive a differentiable and solvable measure for bounding this conditional entropy based on the Gaussian mixture estimation and propose a conditional entropy algorithm to enhance the inversion robustness
  arXiv  Detail & Related papers  (2025-03-01T07:15:21Z)
• Towards safe and tractable Gaussian process-based MPC: Efficient sampling within a sequential quadratic programming framework [35.79393879150088]
  We propose a robust GP-MPC formulation that guarantees constraint satisfaction with high probability. We highlight the improved reachable set approximation compared to existing methods, as well as real-time feasible times.
  arXiv  Detail & Related papers  (2024-09-13T08:15:20Z)
• Monte Carlo Planning for Stochastic Control on Constrained Markov Decision Processes [1.445706856497821]
  This work defines an MDP framework, the textttSD-MDP, where we disentangle the causal structure of MDPs' transition and reward dynamics. We derive theoretical guarantees on the estimation error of the value function under an optimal policy by allowing independent value estimation from Monte Carlo sampling.
  arXiv  Detail & Related papers  (2024-06-23T16:22:40Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond [101.5329678997916]
  We study sample efficient reinforcement learning (RL) under the general framework of interactive decision making. We propose a novel complexity measure, generalized eluder coefficient (GEC), which characterizes the fundamental tradeoff between exploration and exploitation. We show that RL problems with low GEC form a remarkably rich class, which subsumes low Bellman eluder dimension problems, bilinear class, low witness rank problems, PO-bilinear class, and generalized regular PSR.
  arXiv  Detail & Related papers  (2022-11-03T16:42:40Z)
• Online Probabilistic Model Identification using Adaptive Recursive MCMC [8.465242072268019]
  We suggest the Adaptive Recursive Markov Chain Monte Carlo (ARMCMC) method. It eliminates the shortcomings of conventional online techniques while computing the entire probability density function of model parameters. We demonstrate our approach using parameter estimation in a soft bending actuator and the Hunt-Crossley dynamic model.
  arXiv  Detail & Related papers  (2022-10-23T02:06:48Z)
• Modular Conformal Calibration [80.33410096908872]
  We introduce a versatile class of algorithms for recalibration in regression. This framework allows one to transform any regression model into a calibrated probabilistic model. We conduct an empirical study of MCC on 17 regression datasets.
  arXiv  Detail & Related papers  (2022-06-23T03:25:23Z)
• Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes [70.80716221080118]
  The paper takes a generative perspective on policy evaluation via temporal-difference (TD) learning. The OS-GPTD approach is developed to estimate the value function for a given policy by observing a sequence of state-reward pairs. To alleviate the limited expressiveness associated with a single fixed kernel, a weighted ensemble (E) of GP priors is employed to yield an alternative scheme.
  arXiv  Detail & Related papers  (2021-12-01T23:15:09Z)
• Cluster Regularization via a Hierarchical Feature Regression [0.0]
  This paper proposes a novel cluster-based regularization - the hierarchical feature regression (HFR) It mobilizes insights from the domains of machine learning and graph theory to estimate parameters along a supervised hierarchical representation of the predictor set. An application to the prediction of economic growth is used to illustrate the HFR's effectiveness in an empirical setting.
  arXiv  Detail & Related papers  (2021-07-10T13:03:01Z)
• Partial Maximum Correntropy Regression for Robust Trajectory Decoding from Noisy Epidural Electrocorticographic Signals [22.202519467049136]
  The Partial Least Square Regression (PLSR) algorithm exhibits exceptional competence for predicting continuous variables from inter-correlated brain recordings in brain-computer interfaces. The aim of the present study is to propose a robust version of PLSR, namely Partial Maximum Correntropy Regression (PMCR) Compared with the conventional PLSR and the state-of-the-art variant, PMCR realized superior prediction competence on three different performance indicators with contaminated training set.
  arXiv  Detail & Related papers  (2021-06-23T05:22:46Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation [99.92568326314667]
  We propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principle. We demonstrate that ACNML compares favorably to a number of prior techniques for uncertainty estimation in terms of calibration on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-11-05T08:04:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.