BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learning

• URL: http://arxiv.org/abs/2504.09426v1
• Date: Sun, 13 Apr 2025 04:17:12 GMT
• Title: BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learning
• Authors: Shengao Wang, Arjun Chandra, Aoming Liu, Venkatesh Saligrama, Boqing Gong,
• Abstract summary: Human infants rapidly develop visual reasoning skills from minimal input.<n>Recent efforts have leveraged infant-inspired datasets like SAYCam.<n>We propose BabyVLM, a novel framework comprising comprehensive in-domain evaluation benchmarks and a synthetic training dataset.
• Score: 47.451445173060094
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human infants rapidly develop visual reasoning skills from minimal input, suggesting that developmentally inspired pretraining could significantly enhance the efficiency of vision-language models (VLMs). Although recent efforts have leveraged infant-inspired datasets like SAYCam, existing evaluation benchmarks remain misaligned--they are either too simplistic, narrowly scoped, or tailored for large-scale pretrained models. Additionally, training exclusively on infant data overlooks the broader, diverse input from which infants naturally learn. To address these limitations, we propose BabyVLM, a novel framework comprising comprehensive in-domain evaluation benchmarks and a synthetic training dataset created via child-directed transformations of existing datasets. We demonstrate that VLMs trained with our synthetic dataset achieve superior performance on BabyVLM tasks compared to models trained solely on SAYCam or general-purpose data of the SAYCam size. BabyVLM thus provides a robust, developmentally aligned evaluation tool and illustrates how compact models trained on carefully curated data can generalize effectively, opening pathways toward data-efficient vision-language learning paradigms.

Related papers

• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate [118.37653302885607]
  We present the Modality Integration Rate (MIR), an effective, robust, and generalized metric to indicate the multi-modal pre-training quality of Large Vision Language Models (LVLMs) MIR is indicative about training data selection, training strategy schedule, and model architecture design to get better pre-training results.
  arXiv  Detail & Related papers  (2024-10-09T17:59:04Z)
• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress. LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset. Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• The BabyView dataset: High-resolution egocentric videos of infants' and young children's everyday experiences [8.952954042940368]
  We release the largest developmental egocentric video dataset to date -- the BabyView dataset. This 493 hour dataset includes egocentric videos from children spanning 6 months - 5 years of age in both longitudinal, at-home contexts and in a preschool environment. We train self-supervised language and vision models and evaluate their transfer to out-of-distribution tasks including syntactic structure learning, object recognition, depth estimation, and image segmentation.
  arXiv  Detail & Related papers  (2024-06-14T23:52:27Z)
• Data-efficient Large Vision Models through Sequential Autoregression [58.26179273091461]
  We develop an efficient, autoregression-based vision model on a limited dataset. We demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint.
  arXiv  Detail & Related papers  (2024-02-07T13:41:53Z)
• Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning [13.964106147449051]
  Existing solutions concentrate on fine-tuning the pre-trained models on conventional image datasets. We propose a novel and effective framework based on learning Visual Prompts (VPT) in the pre-trained Vision Transformers (ViT) We demonstrate that our new approximations with semantic information are superior to representative capabilities.
  arXiv  Detail & Related papers  (2024-02-04T04:42:05Z)
• On the Automatic Generation and Simplification of Children's Stories [14.465545222216749]
  We first examine the ability of several popular large language models to generate stories with properly adjusted lexical and readability levels. As a second experiment, we explore the ability of state-of-the-art lexical simplification models to generalize to the domain of children's stories. We find that, while the strongest-performing current lexical simplification models do not perform as well on material designed for children due to their reliance on large language models behind the scenes.
  arXiv  Detail & Related papers  (2023-10-27T21:31:34Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.