Training Neural Networks on Data Sources with Unknown Reliability

• URL: http://arxiv.org/abs/2212.02895v4
• Date: Fri, 14 Feb 2025 17:35:40 GMT
• Title: Training Neural Networks on Data Sources with Unknown Reliability
• Authors: Alexander Capstick, Francesca Palermo, Tianyu Cui, Payam Barnaghi,
• Abstract summary: In supervised learning, we aim to train neural networks on each data source for a number of steps proportional to the source's estimated reliability.<n>This way, we allow training on all sources during the warm-up and reduce learning on less reliable sources during the final training stages.<n>We show through diverse experiments that this can significantly improve model performance when trained on mixtures of reliable and unreliable data sources.
• Score: 45.903073137443116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When data is generated by multiple sources, conventional training methods update models assuming equal reliability for each source and do not consider their individual data quality. However, in many applications, sources have varied levels of reliability that can have negative effects on the performance of a neural network. A key issue is that often the quality of the data for individual sources is not known during training. Previous methods for training models in the presence of noisy data do not make use of the additional information that the source label can provide. Focusing on supervised learning, we aim to train neural networks on each data source for a number of steps proportional to the source's estimated reliability by using a dynamic re-weighting strategy motivated by likelihood tempering. This way, we allow training on all sources during the warm-up and reduce learning on less reliable sources during the final training stages, when it has been shown that models overfit to noise. We show through diverse experiments that this can significantly improve model performance when trained on mixtures of reliable and unreliable data sources, and maintain performance when models are trained on reliable sources only.

Related papers

• Source-Free Domain-Invariant Performance Prediction [68.39031800809553]
  We propose a source-free approach centred on uncertainty-based estimation, using a generative model for calibration in the absence of source data. Our experiments on benchmark object recognition datasets reveal that existing source-based methods fall short with limited source sample availability. Our approach significantly outperforms the current state-of-the-art source-free and source-based methods, affirming its effectiveness in domain-invariant performance estimation.
  arXiv  Detail & Related papers  (2024-08-05T03:18:58Z)
• Enhancing Data Quality in Federated Fine-Tuning of Foundation Models [54.757324343062734]
  We propose a data quality control pipeline for federated fine-tuning of foundation models. This pipeline computes scores reflecting the quality of training data and determines a global threshold for a unified standard. Our experiments show that the proposed quality control pipeline facilitates the effectiveness and reliability of the model training, leading to better performance.
  arXiv  Detail & Related papers  (2024-03-07T14:28:04Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Probabilistic Neural Data Fusion for Learning from an Arbitrary Number of Multi-fidelity Data Sets [0.0]
  In this paper, we employ neural networks (NNs) for data fusion in scenarios where data is very scarce. We introduce a unique NN architecture that converts MF modeling into a nonlinear manifold learning problem. Our approach provides a high predictive power while quantifying various sources uncertainties.
  arXiv  Detail & Related papers  (2023-01-30T20:27:55Z)
• Unsupervised Audio Source Separation Using Differentiable Parametric Source Models [8.80867379881193]
  We propose an unsupervised model-based deep learning approach to musical source separation. A neural network is trained to reconstruct the observed mixture as a sum of the sources. The experimental evaluation on a vocal ensemble separation task shows that the proposed method outperforms learning-free methods.
  arXiv  Detail & Related papers  (2022-01-24T11:05:30Z)
• Deep Transfer Learning for Multi-source Entity Linkage via Domain Adaptation [63.24594955429465]
  Multi-source entity linkage is critical in high-impact applications such as data cleaning and user stitching. AdaMEL is a deep transfer learning framework that learns generic high-level knowledge to perform multi-source entity linkage. Our framework achieves state-of-the-art results with 8.21% improvement on average over methods based on supervised learning.
  arXiv  Detail & Related papers  (2021-10-27T15:20:41Z)
• Evaluating the Values of Sources in Transfer Learning [38.93955146476584]
  SEAL-Shap is an efficient source valuation framework for quantifying the usefulness of the sources. Our framework is not only effective in choosing useful transfer sources but also the source values match the intuitive source-target similarity.
  arXiv  Detail & Related papers  (2021-04-26T13:35:24Z)
• Bias-Aware Loss for Training Image and Speech Quality Prediction Models from Multiple Datasets [13.132388683797503]
  We propose a bias-aware loss function that estimates each dataset's biases during training with a linear function. We prove the efficiency of the proposed method by training and validating quality prediction models on synthetic and subjective image and speech quality datasets.
  arXiv  Detail & Related papers  (2021-04-20T19:20:11Z)
• Hidden Biases in Unreliable News Detection Datasets [60.71991809782698]
  We show that selection bias during data collection leads to undesired artifacts in the datasets. We observed a significant drop (>10%) in accuracy for all models tested in a clean split with no train/test source overlap. We suggest future dataset creation include a simple model as a difficulty/bias probe and future model development use a clean non-overlapping site and date split.
  arXiv  Detail & Related papers  (2021-04-20T17:16:41Z)
• Unsupervised Multi-source Domain Adaptation Without Access to Source Data [58.551861130011886]
  Unsupervised Domain Adaptation (UDA) aims to learn a predictor model for an unlabeled domain by transferring knowledge from a separate labeled source domain. We propose a novel and efficient algorithm which automatically combines the source models with suitable weights in such a way that it performs at least as good as the best source model.
  arXiv  Detail & Related papers  (2021-04-05T10:45:12Z)
• Sample-based Regularization: A Transfer Learning Strategy Toward Better Generalization [8.432864879027724]
  Training a deep neural network with a small amount of data is a challenging problem. One of the practical difficulties that we often face is to collect many samples. By using the source model trained with a large-scale dataset, the target model can alleviate the overfitting originated from the lack of training data.
  arXiv  Detail & Related papers  (2020-07-10T06:02:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.