Scaling Laws For Deep Learning Based Image Reconstruction
- URL: http://arxiv.org/abs/2209.13435v1
- Date: Tue, 27 Sep 2022 14:44:57 GMT
- Title: Scaling Laws For Deep Learning Based Image Reconstruction
- Authors: Tobit Klug and Reinhard Heckel
- Abstract summary: We study whether major performance gains are expected from scaling up the training set size.
An initially steep power-law scaling slows significantly already at moderate training set sizes.
We analytically characterize the performance of a linear estimator learned with early stopped gradient descent.
- Score: 26.808569077500128
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep neural networks trained end-to-end to map a measurement of a (noisy)
image to a clean image perform excellent for a variety of linear inverse
problems. Current methods are only trained on a few hundreds or thousands of
images as opposed to the millions of examples deep networks are trained on in
other domains. In this work, we study whether major performance gains are
expected from scaling up the training set size. We consider image denoising,
accelerated magnetic resonance imaging, and super-resolution and empirically
determine the reconstruction quality as a function of training set size, while
optimally scaling the network size. For all three tasks we find that an
initially steep power-law scaling slows significantly already at moderate
training set sizes. Interpolating those scaling laws suggests that even
training on millions of images would not significantly improve performance. To
understand the expected behavior, we analytically characterize the performance
of a linear estimator learned with early stopped gradient descent. The result
formalizes the intuition that once the error induced by learning the signal
model is small relative to the error floor, more training examples do not
improve performance.
Related papers
- Enhancing pretraining efficiency for medical image segmentation via transferability metrics [0.0]
In medical image segmentation tasks, the scarcity of labeled training data poses a significant challenge.
We introduce a novel transferability metric, based on contrastive learning, that measures how robustly a pretrained model is able to represent the target data.
arXiv Detail & Related papers (2024-10-24T12:11:52Z) - A Dynamical Model of Neural Scaling Laws [79.59705237659547]
We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization.
Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
arXiv Detail & Related papers (2024-02-02T01:41:38Z) - Relearning Forgotten Knowledge: on Forgetting, Overfit and Training-Free
Ensembles of DNNs [9.010643838773477]
We introduce a novel score for quantifying overfit, which monitors the forgetting rate of deep models on validation data.
We show that overfit can occur with and without a decrease in validation accuracy, and may be more common than previously appreciated.
We use our observations to construct a new ensemble method, based solely on the training history of a single network, which provides significant improvement without any additional cost in training time.
arXiv Detail & Related papers (2023-10-17T09:22:22Z) - Diffused Redundancy in Pre-trained Representations [98.55546694886819]
We take a closer look at how features are encoded in pre-trained representations.
We find that learned representations in a given layer exhibit a degree of diffuse redundancy.
Our findings shed light on the nature of representations learned by pre-trained deep neural networks.
arXiv Detail & Related papers (2023-05-31T21:00:50Z) - Boosting Verified Training for Robust Image Classifications via
Abstraction [20.656457368486876]
This paper proposes a novel, abstraction-based, certified training method for robust image classifiers.
By training on intervals, all perturbed images that are mapped to the same interval are classified as the same label.
For the abstraction, our training method also enables a sound and complete black-box verification approach.
arXiv Detail & Related papers (2023-03-21T02:38:14Z) - Theoretical Characterization of How Neural Network Pruning Affects its
Generalization [131.1347309639727]
This work makes the first attempt to study how different pruning fractions affect the model's gradient descent dynamics and generalization.
It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero.
More surprisingly, the generalization bound gets better as the pruning fraction gets larger.
arXiv Detail & Related papers (2023-01-01T03:10:45Z) - Slimmable Networks for Contrastive Self-supervised Learning [69.9454691873866]
Self-supervised learning makes significant progress in pre-training large models, but struggles with small models.
We introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers.
A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks.
arXiv Detail & Related papers (2022-09-30T15:15:05Z) - Is Deep Image Prior in Need of a Good Education? [57.3399060347311]
Deep image prior was introduced as an effective prior for image reconstruction.
Despite its impressive reconstructive properties, the approach is slow when compared to learned or traditional reconstruction techniques.
We develop a two-stage learning paradigm to address the computational challenge.
arXiv Detail & Related papers (2021-11-23T15:08:26Z) - Compressive sensing with un-trained neural networks: Gradient descent
finds the smoothest approximation [60.80172153614544]
Un-trained convolutional neural networks have emerged as highly successful tools for image recovery and restoration.
We show that an un-trained convolutional neural network can approximately reconstruct signals and images that are sufficiently structured, from a near minimal number of random measurements.
arXiv Detail & Related papers (2020-05-07T15:57:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.