Finite-sample performance of the maximum likelihood estimator in logistic regression

• URL: http://arxiv.org/abs/2411.02137v1
• Date: Mon, 04 Nov 2024 14:50:15 GMT
• Title: Finite-sample performance of the maximum likelihood estimator in logistic regression
• Authors: Hugo Chardon, Matthieu Lerasle, Jaouad Mourtada,
• Abstract summary: We consider the predictive performance of the maximum likelihood estimator (MLE) for logistic regression. We obtain sharp non-asymptotic guarantees for the existence and excess logistic risk of the MLE.
• Score: 3.7550827441501844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Logistic regression is a classical model for describing the probabilistic dependence of binary responses to multivariate covariates. We consider the predictive performance of the maximum likelihood estimator (MLE) for logistic regression, assessed in terms of logistic risk. We consider two questions: first, that of the existence of the MLE (which occurs when the dataset is not linearly separated), and second that of its accuracy when it exists. These properties depend on both the dimension of covariates and on the signal strength. In the case of Gaussian covariates and a well-specified logistic model, we obtain sharp non-asymptotic guarantees for the existence and excess logistic risk of the MLE. We then generalize these results in two ways: first, to non-Gaussian covariates satisfying a certain two-dimensional margin condition, and second to the general case of statistical learning with a possibly misspecified logistic model. Finally, we consider the case of a Bernoulli design, where the behavior of the MLE is highly sensitive to the parameter direction.

Related papers

• Variational empirical Bayes variable selection in high-dimensional logistic regression [2.4032899110671955]
  We develop a novel and computationally efficient variational approximation thereof. One such novelty is that we develop this approximation directly for the marginal distribution on the model space, rather than on the regression coefficients themselves. We demonstrate the method's strong performance in simulations, and prove that our variational approximation inherits the strong selection consistency property satisfied by the posterior distribution that it is approximating.
  arXiv  Detail & Related papers  (2025-02-14T19:57:13Z)
• High-dimensional logistic regression with missing data: Imputation, regularization, and universality [7.167672851569787]
  We study high-dimensional, ridge-regularized logistic regression. We provide exact characterizations of both the prediction error and the estimation error.
  arXiv  Detail & Related papers  (2024-10-01T21:41:21Z)
• Logistic-beta processes for dependent random probabilities with beta marginals [58.91121576998588]
  We propose a novel process called the logistic-beta process, whose logistic transformation yields a process with common beta marginals. It can model dependence on both discrete and continuous domains, such as space or time, and has a flexible dependence structure through correlation kernels. We illustrate the benefits through nonparametric binary regression and conditional density estimation examples, both in simulation studies and in a pregnancy outcome application.
  arXiv  Detail & Related papers  (2024-02-10T21:41:32Z)
• Nonparametric logistic regression with deep learning [1.2509746979383698]
  In the nonparametric logistic regression, the Kullback-Leibler divergence could diverge easily. Instead of analyzing the excess risk itself, it suffices to show the consistency of the maximum likelihood estimator. As an important application, we derive the convergence rates of the NPMLE with deep neural networks.
  arXiv  Detail & Related papers  (2024-01-23T04:31:49Z)
• Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensions [11.69389391551085]
  This paper investigates the distribution of the maximum-likelihood estimate (MLE) in multinomial logistic models in the high-dimensional regime where dimension and sample size are of the same order.
  arXiv  Detail & Related papers  (2023-05-28T23:33:41Z)
• Statistical Efficiency of Score Matching: The View from Isoperimetry [96.65637602827942]
  We show a tight connection between statistical efficiency of score matching and the isoperimetric properties of the distribution being estimated. We formalize these results both in the sample regime and in the finite regime.
  arXiv  Detail & Related papers  (2022-10-03T06:09:01Z)
• Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiency [59.48096489854697]
  The problem of estimating a linear functional based on observational data is canonical in both the causal inference and bandit literatures. We prove non-asymptotic upper bounds on the mean-squared error of such procedures. We establish its instance-dependent optimality in finite samples via matching non-asymptotic local minimax lower bounds.
  arXiv  Detail & Related papers  (2022-09-26T23:50:55Z)
• On the Double Descent of Random Features Models Trained with SGD [78.0918823643911]
  We study properties of random features (RF) regression in high dimensions optimized by gradient descent (SGD) We derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting. We observe the double descent phenomenon both theoretically and empirically.
  arXiv  Detail & Related papers  (2021-10-13T17:47:39Z)
• Heavy-tailed Streaming Statistical Estimation [58.70341336199497]
  We consider the task of heavy-tailed statistical estimation given streaming $p$ samples. We design a clipped gradient descent and provide an improved analysis under a more nuanced condition on the noise of gradients.
  arXiv  Detail & Related papers  (2021-08-25T21:30:27Z)
• Multivariate Probabilistic Regression with Natural Gradient Boosting [63.58097881421937]
  We propose a Natural Gradient Boosting (NGBoost) approach based on nonparametrically modeling the conditional parameters of the multivariate predictive distribution. Our method is robust, works out-of-the-box without extensive tuning, is modular with respect to the assumed target distribution, and performs competitively in comparison to existing approaches.
  arXiv  Detail & Related papers  (2021-06-07T17:44:49Z)
• The MELODIC family for simultaneous binary logistic regression in a reduced space [0.5330240017302619]
  We propose the MELODIC family for simultaneous binary logistic regression modeling. The model may be interpreted in terms of logistic regression coefficients or in terms of a biplot. Two applications are shown in detail: one relating personality characteristics to drug consumption profiles and one relating personality characteristics to depressive and anxiety disorders.
  arXiv  Detail & Related papers  (2021-02-16T15:47:20Z)
• On the existence of the maximum likelihood estimate and convergence rate under gradient descent for multi-class logistic regression [0.0]
  We show that one method of ensuring its existence is by assigning positive probability to every class in the sample dataset. The notion of data separability is not needed, which is in contrast to the classical set up of multi-class logistic regression.
  arXiv  Detail & Related papers  (2020-12-08T17:21:34Z)
• Two-step penalised logistic regression for multi-omic data with an application to cardiometabolic syndrome [62.997667081978825]
  We implement a two-step approach to multi-omic logistic regression in which variable selection is performed on each layer separately. Our approach should be preferred if the goal is to select as many relevant predictors as possible. Our proposed approach allows us to identify features that characterise cardiometabolic syndrome at the molecular level.
  arXiv  Detail & Related papers  (2020-08-01T10:36:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.