Asyncval: A Toolkit for Asynchronously Validating Dense Retriever Checkpoints during Training

• URL: http://arxiv.org/abs/2202.12510v1
• Date: Fri, 25 Feb 2022 06:07:58 GMT
• Title: Asyncval: A Toolkit for Asynchronously Validating Dense Retriever Checkpoints during Training
• Authors: Shengyao Zhuang and Guido Zuccon
• Abstract summary: A simple strategy to validate deep learning checkpoints is the addition of validation loops to execute during training. The validation of dense retrievers (DR) checkpoints is not as trivial -- and the addition of validation loops is not efficient. We propose Asyncval: a Python-based toolkit for efficiently validating DR checkpoints during training.
• Score: 26.053028706793587
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The process of model checkpoint validation refers to the evaluation of the performance of a model checkpoint executed on a held-out portion of the training data while learning the hyperparameters of the model, and is used to avoid over-fitting and determine when the model has converged so as to stop training. A simple and efficient strategy to validate deep learning checkpoints is the addition of validation loops to execute during training. However, the validation of dense retrievers (DR) checkpoints is not as trivial -- and the addition of validation loops is not efficient. This is because, in order to accurately evaluate the performance of a DR checkpoint, the whole document corpus needs to be encoded into vectors using the current checkpoint before any actual retrieval operation for checkpoint validation can be performed. This corpus encoding process can be very time-consuming if the document corpus contains millions of documents (e.g., 8.8m for MS MARCO and 21m for Natural Questions). Thus, a naive use of validation loops during training will significantly increase training time. To address this issue, in this demo paper, we propose Asyncval: a Python-based toolkit for efficiently validating DR checkpoints during training. Instead of pausing the training loop for validating DR checkpoints, Asyncval decouples the validation loop from the training loop, uses another GPU to automatically validate new DR checkpoints and thus permits to perform validation asynchronously from training. Asyncval also implements a range of different corpus subset sampling strategies for validating DR checkpoints; these strategies allow to further speed up the validation process. We provide an investigation of these methods in terms of their impact on validation time and validation fidelity. Asyncval is made available as an open-source project at \url{https://github.com/ielab/asyncval}.

Related papers

• ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development [9.13331802151585]
  ByteCheckpoint is an industrial-grade checkpointing system for large-scale LFM training. ByteCheckpoint significantly reduces checkpoint stalls, achieving an average reduction of 54.20x. For saving and loading times, ByteCheckpoint achieves improvements of up to 9.96x and 8.80x, respectively.
  arXiv  Detail & Related papers  (2024-07-29T16:18:20Z)
• Fact Checking Beyond Training Set [64.88575826304024]
  We show that the retriever-reader suffers from performance deterioration when it is trained on labeled data from one domain and used in another domain. We propose an adversarial algorithm to make the retriever component robust against distribution shift. We then construct eight fact checking scenarios from these datasets, and compare our model to a set of strong baseline models.
  arXiv  Detail & Related papers  (2024-03-27T15:15:14Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)
• Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training [81.3781338418574]
  We propose relevance-aware contrastive learning. We consistently improve the SOTA unsupervised Contriever model on the BEIR and open-domain QA retrieval benchmarks. Our method can not only beat BM25 after further pre-training on the target corpus but also serves as a good few-shot learner.
  arXiv  Detail & Related papers  (2023-06-05T18:20:27Z)
• Bridging the Training-Inference Gap for Dense Phrase Retrieval [104.4836127502683]
  Building dense retrievers requires a series of standard procedures, including training and validating neural models. In this paper, we explore how the gap between training and inference in dense retrieval can be reduced. We propose an efficient way of validating dense retrievers using a small subset of the entire corpus.
  arXiv  Detail & Related papers  (2022-10-25T00:53:06Z)
• Intersection of Parallels as an Early Stopping Criterion [64.8387564654474]
  We propose a method to spot an early stopping point in the training iterations without the need for a validation set. For a wide range of learning rates, our method, called Cosine-Distance Criterion (CDC), leads to better generalization on average than all the methods that we compare against.
  arXiv  Detail & Related papers  (2022-08-19T19:42:41Z)
• Three New Validators and a Large-Scale Benchmark Ranking for Unsupervised Domain Adaptation [37.03614011735927]
  We propose three new validators for unsupervised domain adaptation (UDA) We compare and rank them against five other existing validators, on a large dataset of 1,000,000 checkpoints. We find that two of our proposed validators achieve state-of-the-art performance in various settings.
  arXiv  Detail & Related papers  (2022-08-15T17:55:26Z)
• Test-Time Adaptation via Self-Training with Nearest Neighbor Information [16.346069386394703]
  Adapting trained classifiers using only online test data is important. One of the popular approaches for test-time adaptation is self-training. We propose a novel test-time adaptation method Test-time Adaptation via Self-Training with nearest neighbor information.
  arXiv  Detail & Related papers  (2022-07-08T05:02:15Z)
• The MultiBERTs: BERT Reproductions for Robustness Analysis [86.29162676103385]
  Re-running pretraining can lead to substantially different conclusions about performance. We introduce MultiBERTs: a set of 25 BERT-base checkpoints. The aim is to enable researchers to draw robust and statistically justified conclusions about pretraining procedures.
  arXiv  Detail & Related papers  (2021-06-30T15:56:44Z)
• Semi-Supervised Learning for Sparsely-Labeled Sequential Data: Application to Healthcare Video Processing [0.8312466807725921]
  We propose a semi-supervised machine learning training strategy to improve event detection performance on sequential data. Our method uses noisy guesses of the events' end times to train event detection models. We show that our strategy outperforms conservative estimates by 12 points of mean average precision for MNIST, and 3.5 points for CIFAR.
  arXiv  Detail & Related papers  (2020-11-28T09:54:44Z)
• Check-N-Run: A Checkpointing System for Training Deep Learning Recommendation Models [5.604501524927757]
  We present Check-N-Run, a scalable checkpointing system for training large machine learning models at Facebook. Check-N-Run uses two primary techniques to address the size and bandwidth challenges. These techniques allow Check-N-Run to reduce the required write bandwidth by 6-17x and the required capacity by 2.5-8x on real-world models.
  arXiv  Detail & Related papers  (2020-10-17T00:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.