Scaling Pre-training to One Hundred Billion Data for Vision Language Models

• URL: http://arxiv.org/abs/2502.07617v1
• Date: Tue, 11 Feb 2025 15:05:33 GMT
• Title: Scaling Pre-training to One Hundred Billion Data for Vision Language Models
• Abstract summary: We investigate the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples.<n>We find that model performance tends to saturate at this scale on many common Western-centric classification and retrieval benchmarks.<n> tasks of cultural diversity achieve more substantial gains from the 100-billion scale web data, thanks to its coverage of long-tail concepts.
• Score: 23.437750698345067
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples. We find that model performance tends to saturate at this scale on many common Western-centric classification and retrieval benchmarks, such as COCO Captions. Nevertheless, tasks of cultural diversity achieve more substantial gains from the 100-billion scale web data, thanks to its coverage of long-tail concepts. Furthermore, we analyze the model's multilinguality and show gains in low-resource languages as well. In addition, we observe that reducing the size of the pretraining dataset via quality filters like using CLIP, typically used to enhance performance, may inadvertently reduce the cultural diversity represented even in large-scale datasets. Our results highlight that while traditional benchmarks may not benefit significantly from scaling noisy, raw web data to 100 billion examples, this data scale is vital for building truly inclusive multimodal systems.

Related papers

• What Can Low Resource Languages Learn From Each Other? [4.131835597238715]
  We propose a data-efficient framework that capitalizes on a cross-script "transfer effect"<n>Our approach first derives language-specific experts from a high-resource base model, then employs task arithmetic to fuse these experts into a unified, high-performance multilingual back- bone.
  arXiv  Detail & Related papers  (2026-08-27T22:32:35Z)
• Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck? [11.243181562664804]
  Large Language Models (LLMs) exhibit a puzzling disparity in their formal linguistic competence.<n>We investigate whether these failures stem from inherent architectural limitations or simply the scarcity of these specific grammatical constructions in web-scale corpora.<n>We pre-train simple GPT-2 Small (124M) models on a 100M-token random sample of the FineWeb corpus and intervene by injecting a minimal amount of synthetic data targeting specific linguistic phenomena.
  arXiv  Detail & Related papers  (2026-04-20T08:11:04Z)
• Curió-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining [12.34636448485891]
  Continued pretraining extends a language model's capabilities by exposing it to additional data, often tailored to a specific linguistic or domain context.<n>We introduce Curi-Edu 7B, a variant trained exclusively on the educational and STEM-filtered subset of the same corpus, totaling just 10 billion tokens.<n>Despite using only 10% of the data and 20% of the computation, Curi-Edu 7B surpasses the full-corpus model in our evaluations.
  arXiv  Detail & Related papers  (2025-12-14T17:19:32Z)
• Retrofitting Small Multilingual Models for Retrieval: Matching 7B Performance with 300M Parameters [30.737678658069097]
  Training effective multilingual embedding models presents unique challenges due to the diversity of languages and task objectives.<n>We investigate key factors that influence the effectiveness of multilingual embeddings, focusing on training data scale, negative sampling strategies, and data diversity.<n>We develop a compact (approximately 300M) multilingual model that achieves retrieval performance comparable to or even surpassing current strong 7B models.
  arXiv  Detail & Related papers  (2025-10-16T03:48:59Z)
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels [96.35283762778137]
  We introduce the Webscale-RL pipeline, a scalable data engine for reinforcement learning.<n>We construct the Webscale-RL dataset, containing 1.2 million examples across more than 9 domains.<n>Our work presents a viable path toward scaling RL to pre-training levels, enabling more capable and efficient language models.
  arXiv  Detail & Related papers  (2025-10-07T22:30:59Z)
• Target-Aware Language Modeling via Granular Data Sampling [25.957424920194914]
  Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. A cost-effective and straightforward approach is sampling with low-dimensional data features. We show that pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B.
  arXiv  Detail & Related papers  (2024-09-23T04:52:17Z)
• Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations [5.065947993017157]
  This study introduces an approach to curate vision-language datasets by employing an image decoding machine learning model. We amassed approximately 9.6 million vision-language paired datasets in VHR imagery. The resultant model outperformed counterparts that did not leverage publicly available vision-language datasets.
  arXiv  Detail & Related papers  (2024-09-11T06:36:08Z)
• CollectiveSFT: Scaling Large Language Models for Chinese Medical Benchmark with Collective Instructions in Healthcare [12.218718086529462]
  This study focuses on the Comprehensive Medical Benchmark in Chinese (CMB) We successfully trained a smaller base model to achieve scores comparable to larger models. By integrating a wide range of instructional content, our approach addresses potential issues such as data quality inconsistencies.
  arXiv  Detail & Related papers  (2024-07-29T05:00:48Z)
• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance [68.18779562801762]
  multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance. Our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.
  arXiv  Detail & Related papers  (2024-04-04T17:58:02Z)
• Data Engineering for Scaling Language Models to 128K Context [98.41554785106902]
  We study the continual pretraining recipe for scaling language models' context lengths to 128K. We find that naively upsampling longer data on certain domains like books, a common practice of existing work, gives suboptimal performance. Our recipe outperforms strong open-source long-context models and closes the gap to frontier models like GPT-4 128K.
  arXiv  Detail & Related papers  (2024-02-15T18:19:16Z)
• Delving Deeper into Data Scaling in Masked Image Modeling [145.36501330782357]
  We conduct an empirical study on the scaling capability of masked image modeling (MIM) methods for visual recognition. Specifically, we utilize the web-collected Coyo-700M dataset. Our goal is to investigate how the performance changes on downstream tasks when scaling with different sizes of data and models.
  arXiv  Detail & Related papers  (2023-05-24T15:33:46Z)
• Prefix Language Models are Unified Modal Learners [30.666873206462295]
  We show that a unified modal model could be learned with a prefix language modeling objective upon text and image sequences. Thanks to the simple but powerful pre-training paradigm, our proposed model, DaVinci, is simple to train, scalable to huge data, and adaptable to a variety of downstream tasks.
  arXiv  Detail & Related papers  (2022-06-15T17:49:38Z)
• Wukong: 100 Million Large-scale Chinese Cross-modal Pre-training Dataset and A Foundation Framework [99.38817546900405]
  This paper presents a large-scale Chinese cross-modal dataset for benchmarking different multi-modal pre-training methods. We release a Large-Scale Chinese Cross-modal dataset named Wukong, containing 100 million Chinese image-text pairs from the web.
  arXiv  Detail & Related papers  (2022-02-14T14:37:15Z)
• Improving Classifier Training Efficiency for Automatic Cyberbullying Detection with Feature Density [58.64907136562178]
  We study the effectiveness of Feature Density (FD) using different linguistically-backed feature preprocessing methods. We hypothesise that estimating dataset complexity allows for the reduction of the number of required experiments. The difference in linguistic complexity of datasets allows us to additionally discuss the efficacy of linguistically-backed word preprocessing.
  arXiv  Detail & Related papers  (2021-11-02T15:48:28Z)
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition [126.5605160882849]
  We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency. We report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks.
  arXiv  Detail & Related papers  (2021-09-27T17:59:19Z)
• Does language help generalization in vision models? [0.0]
  We show that a visual model trained on a very large supervised image dataset (ImageNet-21k) can be as efficient for generalization as its multimodal counterpart (CLIP) When compared to other standard visual or language models, the latent representations of BiT-M were found to be just as "linguistic" as those of CLIP.
  arXiv  Detail & Related papers  (2021-04-16T18:54:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.