Data-Efficient Challenges in Visual Inductive Priors: A Retrospective

• URL: http://arxiv.org/abs/2506.08612v1
• Date: Tue, 10 Jun 2025 09:21:48 GMT
• Title: Data-Efficient Challenges in Visual Inductive Priors: A Retrospective
• Authors: Robert-Jan Bruintjes, Attila Lengyel, Osman Semih Kayhan, Davide Zambrano, Nergis Tömen, Hadi Jamali-Rad, Jan van Gemert,
• Abstract summary: Deep Learning requires large amounts of data to train models that work well.<n>In data-deficient settings, performance can be degraded.<n>We investigate which Deep Learning methods benefit training models in a data-deficient setting.
• Score: 9.961131337487243
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Learning requires large amounts of data to train models that work well. In data-deficient settings, performance can be degraded. We investigate which Deep Learning methods benefit training models in a data-deficient setting, by organizing the "VIPriors: Visual Inductive Priors for Data-Efficient Deep Learning" workshop series, featuring four editions of data-impaired challenges. These challenges address the problem of training deep learning models for computer vision tasks with limited data. Participants are limited to training models from scratch using a low number of training samples and are not allowed to use any form of transfer learning. We aim to stimulate the development of novel approaches that incorporate prior knowledge to improve the data efficiency of deep learning models. Successful challenge entries make use of large model ensembles that mix Transformers and CNNs, as well as heavy data augmentation. Novel prior knowledge-based methods contribute to success in some entries.

Related papers

• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress.<n>LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset.<n>Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• VIPriors 4: Visual Inductive Priors for Data-Efficient Deep Learning Challenges [12.615348941903594]
  Fourth edition of the "VIPriors: Visual Inductive Priors for Data-Efficient Deep Learning" workshop features two data-impaired challenges. These challenges address the problem of training deep learning models for computer vision tasks with limited data. We aim to stimulate the development of novel approaches that incorporate inductive biases to improve the data efficiency of deep learning models.
  arXiv  Detail & Related papers  (2024-06-26T08:50:51Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection [56.292071534857946]
  Recent data-privacy laws have sparked interest in machine unlearning. Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset. We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU) We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
  arXiv  Detail & Related papers  (2023-12-07T07:17:24Z)
• VIPriors 3: Visual Inductive Priors for Data-Efficient Deep Learning Challenges [13.085098213230568]
  Third edition of the "VIPriors: Visual Inductive Priors for Data-Efficient Deep Learning" workshop featured four data-impaired challenges. Challenges focused on addressing the limitations of data availability in training deep learning models for computer vision tasks.
  arXiv  Detail & Related papers  (2023-05-31T09:31:54Z)
• Phased Data Augmentation for Training a Likelihood-Based Generative Model with Limited Data [0.0]
  Generative models excel in creating realistic images, yet their dependency on extensive datasets for training presents significant challenges. Current data-efficient methods largely focus on GAN architectures, leaving a gap in training other types of generative models. "phased data augmentation" is a novel technique that addresses this gap by optimizing training in limited data scenarios without altering the inherent data distribution.
  arXiv  Detail & Related papers  (2023-05-22T03:38:59Z)
• Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability [53.27240222619834]
  Knowledge Distillation as Efficient Pre-training aims to efficiently transfer the learned feature representation from pre-trained models to new student models for future downstream tasks. Our method performs comparably with supervised pre-training counterparts in 3 downstream tasks and 9 downstream datasets requiring 10x less data and 5x less pre-training time.
  arXiv  Detail & Related papers  (2022-03-10T06:23:41Z)
• VIPriors 2: Visual Inductive Priors for Data-Efficient Deep Learning Challenges [13.085098213230568]
  Second edition of "VIPriors: Visual Inductive Priors for Data-Efficient Deep Learning" challenges. Models are trained from scratch on a reduced number of training samples for various key computer vision tasks. Results: The provided baselines are outperformed by a large margin in all five challenges.
  arXiv  Detail & Related papers  (2022-01-21T10:20:52Z)
• Training Deep Networks from Zero to Hero: avoiding pitfalls and going beyond [59.94347858883343]
  This tutorial covers the basic steps as well as more recent options to improve models. It can be particularly useful in datasets that are not as well-prepared as those in challenges.
  arXiv  Detail & Related papers  (2021-09-06T21:31:42Z)
• VIPriors 1: Visual Inductive Priors for Data-Efficient Deep Learning Challenges [8.50468505606714]
  We offer four data-impaired challenges, where models are trained from scratch, and we reduce the number of training samples to a fraction of the full set. To encourage data efficient solutions, we prohibited the use of pre-trained models and other transfer learning techniques.
  arXiv  Detail & Related papers  (2021-03-05T15:58:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.