Regularized Modal Regression on Markov-dependent Observations: A Theoretical Assessment

• URL: http://arxiv.org/abs/2112.04779v1
• Date: Thu, 9 Dec 2021 09:08:52 GMT
• Title: Regularized Modal Regression on Markov-dependent Observations: A Theoretical Assessment
• Authors: Tielang Gong, Yuxin Dong, Hong Chen, Bo Dong, Wei Feng, Chen Li
• Abstract summary: This paper concerns the statistical property of regularized modal regression (RMR) within an important dependence structure - Markov dependent. We establish the upper bound for RMR estimator under moderate conditions and give an explicit learning rate. Our results show that the Markov dependence impacts on the generalization error in the way that sample size would be discounted by a multiplicative factor depending on the spectral gap of underlying Markov chain.
• Score: 13.852720406291875
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modal regression, a widely used regression protocol, has been extensively investigated in statistical and machine learning communities due to its robustness to outliers and heavy-tailed noises. Understanding modal regression's theoretical behavior can be fundamental in learning theory. Despite significant progress in characterizing its statistical property, the majority of the results are based on the assumption that samples are independent and identical distributed (i.i.d.), which is too restrictive for real-world applications. This paper concerns the statistical property of regularized modal regression (RMR) within an important dependence structure - Markov dependent. Specifically, we establish the upper bound for RMR estimator under moderate conditions and give an explicit learning rate. Our results show that the Markov dependence impacts on the generalization error in the way that sample size would be discounted by a multiplicative factor depending on the spectral gap of underlying Markov chain. This result shed a new light on characterizing the theoretical underpinning for robust regression.

Related papers

• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• Statistical Robustness of Interval CVaR Based Regression Models under Perturbation and Contamination [1.578201299411112]
  We address the robust nonlinear regression based on the so-called interval conditional value-at-risk (In-CVaR)<n>We rigorously quantify robustness under contamination, with a unified study of distributional breakdown point for a broad class of regression models.<n>We show that the In-CVaR based estimator is qualitatively robust in terms of the Prokhorov metric if and only if the largest portion of losses is trimmed.
  arXiv  Detail & Related papers  (2026-01-16T16:41:57Z)
• Physics-Informed Singular-Value Learning for Cross-Covariances Forecasting in Financial Markets [0.0]
  We design a randommatrix-inspired neural architecture that learns a nonlinear mapping from empirical singular values to their corresponding cleaned values.<n>By construction, the network can recover the analytical solution as a special case, yet it remains flexible enough to adapt to non-stationary dynamics and mode-driven distortions.
  arXiv  Detail & Related papers  (2026-01-12T16:18:08Z)
• Uncertainty Quantification for Regression using Proper Scoring Rules [76.24649098854219]
  We introduce a unified UQ framework for regression based on proper scoring rules, such as CRPS, logarithmic, squared error, and quadratic scores.<n>We derive closed-form expressions for the uncertainty measures under practical parametric assumptions and show how to estimate them using ensembles of models.<n>Our broad evaluation on synthetic and real-world regression datasets provides guidance for selecting reliable UQ measures.
  arXiv  Detail & Related papers  (2025-09-30T17:52:12Z)
• Risk Bounds For Distributional Regression [9.92024586772767]
  General upper bounds are established for the continuous ranked score (CRPS) and the worst-case mean squared error (MSE) across the domain.<n>Experiments on both simulated and real data validate the theoretical contributions, demonstrating their practical effectiveness.
  arXiv  Detail & Related papers  (2025-05-14T02:22:12Z)
• Global Convergence of Continual Learning on Non-IID Data [51.99584235667152]
  We provide a general and comprehensive theoretical analysis for continual learning of regression models. We establish the almost sure convergence results of continual learning under a general data condition for the first time.
  arXiv  Detail & Related papers  (2025-03-24T10:06:07Z)
• Uncertainty quantification for Markov chains with application to temporal difference learning [63.49764856675643]
  We develop novel high-dimensional concentration inequalities and Berry-Esseen bounds for vector- and matrix-valued functions of Markov chains. We analyze the TD learning algorithm, a widely used method for policy evaluation in reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-19T15:33:55Z)
• Bias Amplification: Large Language Models as Increasingly Biased Media [12.376194654498383]
  We introduce a benchmark specifically designed to measure political bias amplification in Large Language Models (LLMs)<n>Our empirical study using GPT-2 reveals consistent and substantial political bias intensification over iterative synthetic training cycles.<n>We evaluate three mitigation strategies, Overfitting, Preservation, and Accumulation, and demonstrate that bias amplification persists independently of model collapse.
  arXiv  Detail & Related papers  (2024-10-19T22:53:27Z)
• Statistical Agnostic Regression: a machine learning method to validate regression models [0.0]
  We introduce Statistical Agnostic Regression (SAR) for evaluating the statistical significance of machine learning (ML)-based linear regression models. We define a threshold that ensures there is sufficient evidence, with a probability of at least $1-eta$, to conclude the existence of a linear relationship in the population between the explanatory (feature) and the response (label) variables.
  arXiv  Detail & Related papers  (2024-02-23T09:19:26Z)
• Trade-off Between Dependence and Complexity for Nonparametric Learning -- an Empirical Process Approach [10.27974860479791]
  In many applications where the data exhibit temporal dependencies, the corresponding empirical processes are much less understood. We present a general bound on the expected supremum of empirical processes under standard $beta/rho$-mixing assumptions. We show that even under long-range dependence, it is possible to attain the same rates as in the i.i.d. setting.
  arXiv  Detail & Related papers  (2024-01-17T05:08:37Z)
• A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning [129.63326990812234]
  We propose a technique named data-dependent contraction to capture how modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment.
  arXiv  Detail & Related papers  (2023-10-07T09:15:08Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• On Regression in Extreme Regions [0.7974430263940756]
  We establish a statistical learning theoretical framework aimed at extrapolation, or out-of-domain generalization, on the unobserved tails of covariates.<n>We address the stylized problem of nonparametric least squares regression with predictors chosen from a Vapnik-Chervonenkis class.<n>We quantify the predictive performance on tail regions in terms of excess risk, presenting it as a finite sample risk bound with a clear bias-variance decomposition.
  arXiv  Detail & Related papers  (2023-03-06T12:55:38Z)
• On LASSO for High Dimensional Predictive Regression [0.0]
  This paper examines LASSO, a widely-used $L_1$-penalized regression method, in high dimensional linear predictive regressions. The consistency of LASSO is contingent upon two key components: the deviation bound of the cross product of the regressors and the error term. Using machine learning and macroeconomic domain expertise, LASSO demonstrates strong performance in forecasting the unemployment rate.
  arXiv  Detail & Related papers  (2022-12-14T06:14:58Z)
• Are Latent Factor Regression and Sparse Regression Adequate? [0.49416305961918056]
  We provide theoretical guarantees for the estimation of our model under the existence of sub-Gaussian and heavy-tailed noises. We propose the Factor-Adjusted de-Biased Test (FabTest) and a two-stage ANOVA type test respectively. Numerical results illustrate the robustness and effectiveness of our model against latent factor regression and sparse linear regression models.
  arXiv  Detail & Related papers  (2022-03-02T16:22:23Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)
• Inference for Network Regression Models with Community Structure [1.7188280334580197]
  We present a novel regression modeling framework that models the errors as resulting from a community-based dependence structure. We exploit the subsequent exchangeability properties of the error distribution to obtain parsimonious standard errors for regression parameters.
  arXiv  Detail & Related papers  (2021-06-08T12:04:31Z)
• Counterfactual Maximum Likelihood Estimation for Training Deep Networks [83.44219640437657]
  Deep learning models are prone to learning spurious correlations that should not be learned as predictive clues. We propose a causality-based training framework to reduce the spurious correlations caused by observable confounders. We conduct experiments on two real-world tasks: Natural Language Inference (NLI) and Image Captioning.
  arXiv  Detail & Related papers  (2021-06-07T17:47:16Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• GenDICE: Generalized Offline Estimation of Stationary Values [108.17309783125398]
  We show that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions. The resulting algorithm, GenDICE, is straightforward and effective.
  arXiv  Detail & Related papers  (2020-02-21T00:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.