Statistical Limits of Adaptive Linear Models: Low-Dimensional Estimation and Inference

• URL: http://arxiv.org/abs/2310.00532v2
• Date: Sat, 28 Oct 2023 13:54:57 GMT
• Title: Statistical Limits of Adaptive Linear Models: Low-Dimensional Estimation and Inference
• Authors: Licong Lin, Mufang Ying, Suvrojit Ghosh, Koulik Khamaru, Cun-Hui Zhang
• Abstract summary: We show that the error of estimating a single coordinate can be enlarged by a multiple of $sqrtd$ when data are allowed to be arbitrarily adaptive. We propose a novel estimator for single coordinate inference via solving a Two-stage Adaptive Linear Estimating equation (TALE)
• Score: 5.924780594614676
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimation and inference in statistics pose significant challenges when data are collected adaptively. Even in linear models, the Ordinary Least Squares (OLS) estimator may fail to exhibit asymptotic normality for single coordinate estimation and have inflated error. This issue is highlighted by a recent minimax lower bound, which shows that the error of estimating a single coordinate can be enlarged by a multiple of $\sqrt{d}$ when data are allowed to be arbitrarily adaptive, compared with the case when they are i.i.d. Our work explores this striking difference in estimation performance between utilizing i.i.d. and adaptive data. We investigate how the degree of adaptivity in data collection impacts the performance of estimating a low-dimensional parameter component in high-dimensional linear models. We identify conditions on the data collection mechanism under which the estimation error for a low-dimensional parameter component matches its counterpart in the i.i.d. setting, up to a factor that depends on the degree of adaptivity. We show that OLS or OLS on centered data can achieve this matching error. In addition, we propose a novel estimator for single coordinate inference via solving a Two-stage Adaptive Linear Estimating equation (TALE). Under a weaker form of adaptivity in data collection, we establish an asymptotic normality property of the proposed estimator.

Related papers

• Precise Model Benchmarking with Only a Few Observations [6.092112060364272]
  We propose an empirical Bayes (EB) estimator that balances direct and regression estimates for each subgroup separately. EB consistently provides more precise estimates of the LLM performance compared to the direct and regression approaches.
  arXiv  Detail & Related papers  (2024-10-07T17:26:31Z)
• Efficient adjustment for complex covariates: Gaining efficiency with DOPE [56.537164957672715]
  We propose a framework that accommodates adjustment for any subset of information expressed by the covariates. Based on our theoretical results, we propose the Debiased Outcome-adapted Propensity Estorimator (DOPE) for efficient estimation of the average treatment effect (ATE) Our results show that the DOPE provides an efficient and robust methodology for ATE estimation in various observational settings.
  arXiv  Detail & Related papers  (2024-02-20T13:02:51Z)
• Leveraging Variational Autoencoders for Parameterized MMSE Estimation [10.141454378473972]
  We propose a variational autoencoder-based framework for parameterizing a conditional linear minimum mean squared error estimator. The derived estimator is shown to approximate the minimum mean squared error estimator by utilizing the variational autoencoder as a generative prior for the estimation problem. We conduct a rigorous analysis by bounding the difference between the proposed and the minimum mean squared error estimator.
  arXiv  Detail & Related papers  (2023-07-11T15:41:34Z)
• Semi-Supervised Quantile Estimation: Robust and Efficient Inference in High Dimensional Settings [0.5735035463793009]
  We consider quantile estimation in a semi-supervised setting, characterized by two available data sets. We propose a family of semi-supervised estimators for the response quantile(s) based on the two data sets.
  arXiv  Detail & Related papers  (2022-01-25T10:02:23Z)
• Estimation of Local Average Treatment Effect by Data Combination [3.655021726150368]
  It is important to estimate the local average treatment effect (LATE) when compliance with a treatment assignment is incomplete. Previously proposed methods for LATE estimation required all relevant variables to be jointly observed in a single dataset. We propose a weighted least squares estimator that enables simpler model selection by avoiding the minimax objective formulation.
  arXiv  Detail & Related papers  (2021-09-11T03:51:48Z)
• Near-optimal inference in adaptive linear regression [60.08422051718195]
  Even simple methods like least squares can exhibit non-normal behavior when data is collected in an adaptive manner. We propose a family of online debiasing estimators to correct these distributional anomalies in at least squares estimation. We demonstrate the usefulness of our theory via applications to multi-armed bandit, autoregressive time series estimation, and active learning with exploration.
  arXiv  Detail & Related papers  (2021-07-05T21:05:11Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)
• Asymptotic Analysis of an Ensemble of Randomly Projected Linear Discriminants [94.46276668068327]
  In [1], an ensemble of randomly projected linear discriminants is used to classify datasets. We develop a consistent estimator of the misclassification probability as an alternative to the computationally-costly cross-validation estimator. We also demonstrate the use of our estimator for tuning the projection dimension on both real and synthetic data.
  arXiv  Detail & Related papers  (2020-04-17T12:47:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.