Memory Efficient Meta-Learning with Large Images

• URL: http://arxiv.org/abs/2107.01105v1
• Date: Fri, 2 Jul 2021 14:37:13 GMT
• Title: Memory Efficient Meta-Learning with Large Images
• Authors: John Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann, Sebastian Nowozin, Richard E. Turner
• Abstract summary: Meta learning approaches to few-shot classification are computationally efficient at test time requiring just a few optimization steps or single forward pass to learn a new task. This limitation arises because a task's entire support set, which can contain up to 1000 images, must be processed before an optimization step can be taken. We propose LITE, a general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU.
• Score: 62.70515410249566
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Meta learning approaches to few-shot classification are computationally efficient at test time requiring just a few optimization steps or single forward pass to learn a new task, but they remain highly memory-intensive to train. This limitation arises because a task's entire support set, which can contain up to 1000 images, must be processed before an optimization step can be taken. Harnessing the performance gains offered by large images thus requires either parallelizing the meta-learner across multiple GPUs, which may not be available, or trade-offs between task and image size when memory constraints apply. We improve on both options by proposing LITE, a general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU. We achieve this by observing that the gradients for a task can be decomposed into a sum of gradients over the task's training images. This enables us to perform a forward pass on a task's entire training set but realize significant memory savings by back-propagating only a random subset of these images which we show is an unbiased approximation of the full gradient. We use LITE to train meta-learners and demonstrate new state-of-the-art accuracy on the real-world ORBIT benchmark and 3 of the 4 parts of the challenging VTAB+MD benchmark relative to leading meta-learners. LITE also enables meta-learners to be competitive with transfer learning approaches but at a fraction of the test-time computational cost, thus serving as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.

Related papers

• Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning [12.5354658533836]
  Humans possess remarkable ability to accurately classify new, unseen images after being exposed to only a few examples. For artificial neural network models, determining the most relevant features for distinguishing between two images with limited samples presents a challenge. We propose an intra-task mutual attention method for few-shot learning, that involves splitting the support and query samples into patches.
  arXiv  Detail & Related papers  (2024-05-06T02:02:57Z)
• ELIP: Efficient Language-Image Pre-training with Fewer Vision Tokens [75.09406436851445]
  We propose a vision token pruning and merging method ELIP, to remove less influential tokens based on the supervision of language outputs. Our experiments demonstrate that with the removal of 30$%$ vision tokens across 12 ViT layers, ELIP maintains significantly comparable performance.
  arXiv  Detail & Related papers  (2023-09-28T05:31:07Z)
• FastMIM: Expediting Masked Image Modeling Pre-training for Vision [65.47756720190155]
  FastMIM is a framework for pre-training vision backbones with low-resolution input images. It reconstructs Histograms of Oriented Gradients (HOG) feature instead of original RGB values of the input images. It can achieve 83.8%/84.1% top-1 accuracy on ImageNet-1K with ViT-B/Swin-B as backbones.
  arXiv  Detail & Related papers  (2022-12-13T14:09:32Z)
• EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones [80.662250618795]
  This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers) As an off-the-shelf method, it reduces the wall-time training cost of a wide variety of popular models by >1.5x on ImageNet-1K/22K without sacrificing accuracy.
  arXiv  Detail & Related papers  (2022-11-17T17:38:55Z)
• End-to-end Multiple Instance Learning with Gradient Accumulation [2.2612425542955292]
  We propose a training strategy that enables end-to-end training of ABMIL models without being limited by GPU memory. We conduct experiments on both QMNIST and Imagenette to investigate the performance and training time. This memory-efficient approach, although slower, reaches performance indistinguishable from the memory-expensive baseline.
  arXiv  Detail & Related papers  (2022-03-08T10:14:51Z)
• Task Attended Meta-Learning for Few-Shot Learning [3.0724051098062097]
  We introduce a training curriculum motivated by selective focus in humans, called task attended meta-training, to weight the tasks in a batch. The comparisons of the models with their non-task-attended counterparts on complex datasets validate its effectiveness.
  arXiv  Detail & Related papers  (2021-06-20T07:34:37Z)
• Few-Shot Learning for Image Classification of Common Flora [0.0]
  We will showcase our results from testing various state-of-the-art transfer learning weights and architectures versus similar state-of-the-art works in the meta-learning field for image classification utilizing Model-Agnostic Meta Learning (MAML) Our results show that both practices provide adequate performance when the dataset is sufficiently large, but that they both also struggle when data sparsity is introduced to maintain sufficient performance.
  arXiv  Detail & Related papers  (2021-05-07T03:54:51Z)
• Data Augmentation for Meta-Learning [58.47185740820304]
  meta-learning algorithms sample data, query data, and tasks on each training step. Data augmentation can be used not only to expand the number of images available per class, but also to generate entirely new classes/tasks. Our proposed meta-specific data augmentation significantly improves the performance of meta-learners on few-shot classification benchmarks.
  arXiv  Detail & Related papers  (2020-10-14T13:48:22Z)
• Expert Training: Task Hardness Aware Meta-Learning for Few-Shot Classification [62.10696018098057]
  We propose an easy-to-hard expert meta-training strategy to arrange the training tasks properly. A task hardness aware module is designed and integrated into the training procedure to estimate the hardness of a task. Experimental results on the miniImageNet and tieredImageNetSketch datasets show that the meta-learners can obtain better results with our expert training strategy.
  arXiv  Detail & Related papers  (2020-07-13T08:49:00Z)
• Task Augmentation by Rotating for Meta-Learning [5.646772123578524]
  We introduce a task augmentation method by rotating, which increases the number of classes by rotating the original images 90, 180 and 270 degrees. Experimental results show that our approach is better than the rotation for increasing the number of images and state-of-the-art performance on miniImageNet, CIFAR-FS, and FC100 few-shot learning benchmarks.
  arXiv  Detail & Related papers  (2020-02-08T07:57:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.