Convergence Rates for Learning Linear Operators from Noisy Data
- URL: http://arxiv.org/abs/2108.12515v1
- Date: Fri, 27 Aug 2021 22:09:53 GMT
- Title: Convergence Rates for Learning Linear Operators from Noisy Data
- Authors: Maarten V. de Hoop, Nikola B. Kovachki, Nicholas H. Nelsen, Andrew M.
Stuart
- Abstract summary: We study the inverse problem of learning a linear operator on a space from its noisy pointwise evaluations on random input data.
We establish posterior contraction rates with respect to a family of Bochner norms as the number of data tend to infinity lower on the estimation error.
These convergence rates highlight and quantify the difficulty of learning linear operators in comparison with the learning of bounded or compact ones.
- Score: 6.4423565043274795
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We study the Bayesian inverse problem of learning a linear operator on a
Hilbert space from its noisy pointwise evaluations on random input data. Our
framework assumes that this target operator is self-adjoint and diagonal in a
basis shared with the Gaussian prior and noise covariance operators arising
from the imposed statistical model and is able to handle target operators that
are compact, bounded, or even unbounded. We establish posterior contraction
rates with respect to a family of Bochner norms as the number of data tend to
infinity and derive related lower bounds on the estimation error. In the large
data limit, we also provide asymptotic convergence rates of suitably defined
excess risk and generalization gap functionals associated with the posterior
mean point estimator. In doing so, we connect the posterior consistency results
to nonparametric learning theory. Furthermore, these convergence rates
highlight and quantify the difficulty of learning unbounded linear operators in
comparison with the learning of bounded or compact ones. Numerical experiments
confirm the theory and demonstrate that similar conclusions may be expected in
more general problem settings.
Related papers
- A Statistical Theory of Regularization-Based Continual Learning [10.899175512941053]
We provide a statistical analysis of regularization-based continual learning on a sequence of linear regression tasks.
We first derive the convergence rate for the oracle estimator obtained as if all data were available simultaneously.
A byproduct of our theoretical analysis is the equivalence between early stopping and generalized $ell$-regularization.
arXiv Detail & Related papers (2024-06-10T12:25:13Z) - Learning Operators with Stochastic Gradient Descent in General Hilbert
Spaces [6.690174942973101]
This study investigates leveraging gradient descent (SGD) to learn operators between general Hilbert spaces.
We establish upper bounds for convergence rates of the SGD algorithm and conduct a minimax lower bound analysis.
Applying our analysis to operator learning problems based on vector-valued and real-valued kernel Hilbert spaces yields new convergence results.
arXiv Detail & Related papers (2024-02-07T09:31:01Z) - Learning a Gaussian Mixture for Sparsity Regularization in Inverse
Problems [2.375943263571389]
In inverse problems, the incorporation of a sparsity prior yields a regularization effect on the solution.
We propose a probabilistic sparsity prior formulated as a mixture of Gaussians, capable of modeling sparsity with respect to a generic basis.
We put forth both a supervised and an unsupervised training strategy to estimate the parameters of this network.
arXiv Detail & Related papers (2024-01-29T22:52:57Z) - Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training.
We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
arXiv Detail & Related papers (2023-10-20T12:45:12Z) - Kernel-based off-policy estimation without overlap: Instance optimality
beyond semiparametric efficiency [53.90687548731265]
We study optimal procedures for estimating a linear functional based on observational data.
For any convex and symmetric function class $mathcalF$, we derive a non-asymptotic local minimax bound on the mean-squared error.
arXiv Detail & Related papers (2023-01-16T02:57:37Z) - Instance-Dependent Generalization Bounds via Optimal Transport [51.71650746285469]
Existing generalization bounds fail to explain crucial factors that drive the generalization of modern neural networks.
We derive instance-dependent generalization bounds that depend on the local Lipschitz regularity of the learned prediction function in the data space.
We empirically analyze our generalization bounds for neural networks, showing that the bound values are meaningful and capture the effect of popular regularization methods during training.
arXiv Detail & Related papers (2022-11-02T16:39:42Z) - Learning Dynamical Systems via Koopman Operator Regression in
Reproducing Kernel Hilbert Spaces [52.35063796758121]
We formalize a framework to learn the Koopman operator from finite data trajectories of the dynamical system.
We link the risk with the estimation of the spectral decomposition of the Koopman operator.
Our results suggest RRR might be beneficial over other widely used estimators.
arXiv Detail & Related papers (2022-05-27T14:57:48Z) - Optimal variance-reduced stochastic approximation in Banach spaces [114.8734960258221]
We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space.
We establish non-asymptotic bounds for both the operator defect and the estimation error.
arXiv Detail & Related papers (2022-01-21T02:46:57Z) - Nonparametric Score Estimators [49.42469547970041]
Estimating the score from a set of samples generated by an unknown distribution is a fundamental task in inference and learning of probabilistic models.
We provide a unifying view of these estimators under the framework of regularized nonparametric regression.
We propose score estimators based on iterative regularization that enjoy computational benefits from curl-free kernels and fast convergence.
arXiv Detail & Related papers (2020-05-20T15:01:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.