Progress in Self-Certified Neural Networks

• URL: http://arxiv.org/abs/2111.07737v1
• Date: Mon, 15 Nov 2021 13:39:44 GMT
• Title: Progress in Self-Certified Neural Networks
• Authors: Maria Perez-Ortiz, Omar Rivasplata, Emilio Parrado-Hernandez, Benjamin Guedj, John Shawe-Taylor
• Abstract summary: A learning method is self-certified if it uses all available data to simultaneously learn a predictor and certify its quality. Recent work has shown that neural network models trained by optimising PAC-Bayes bounds lead to accurate predictors. We show that in data starvation regimes, holding out data for the test set bounds adversely affects generalisation performance.
• Score: 13.434562713466246
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A learning method is self-certified if it uses all available data to simultaneously learn a predictor and certify its quality with a statistical certificate that is valid on unseen data. Recent work has shown that neural network models trained by optimising PAC-Bayes bounds lead not only to accurate predictors, but also to tight risk certificates, bearing promise towards achieving self-certified learning. In this context, learning and certification strategies based on PAC-Bayes bounds are especially attractive due to their ability to leverage all data to learn a posterior and simultaneously certify its risk. In this paper, we assess the progress towards self-certification in probabilistic neural networks learnt by PAC-Bayes inspired objectives. We empirically compare (on 4 classification datasets) classical test set bounds for deterministic predictors and a PAC-Bayes bound for randomised self-certified predictors. We first show that both of these generalisation bounds are not too far from out-of-sample test set errors. We then show that in data starvation regimes, holding out data for the test set bounds adversely affects generalisation performance, while self-certified strategies based on PAC-Bayes bounds do not suffer from this drawback, proving that they might be a suitable choice for the small data regime. We also find that probabilistic neural networks learnt by PAC-Bayes inspired objectives lead to certificates that can be surprisingly competitive with commonly used test set bounds.

Related papers

• FullCert: Deterministic End-to-End Certification for Training and Inference of Neural Networks [62.897993591443594]
  FullCert is the first end-to-end certifier with sound, deterministic bounds. We experimentally demonstrate FullCert's feasibility on two datasets.
  arXiv  Detail & Related papers  (2024-06-17T13:23:52Z)
• Robust Deep Learning for Autonomous Driving [0.0]
  We introduce a new criterion to reliably estimate model confidence: the true class probability ( TCP) Since the true class is by essence unknown at test time, we propose to learn TCP criterion from data with an auxiliary model, introducing a specific learning scheme adapted to this context. We tackle the challenge of jointly detecting misclassification and out-of-distributions samples by introducing a new uncertainty measure based on evidential models and defined on the simplex.
  arXiv  Detail & Related papers  (2022-11-14T22:07:11Z)
• Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
  We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
  arXiv  Detail & Related papers  (2022-01-26T19:59:14Z)
• Knowledge-driven Active Learning [70.37119719069499]
  Active learning strategies aim at minimizing the amount of labelled data required to train a Deep Learning model. Most active strategies are based on uncertain sample selection, and even often restricted to samples lying close to the decision boundary. Here we propose to take into consideration common domain-knowledge and enable non-expert users to train a model with fewer samples.
  arXiv  Detail & Related papers  (2021-10-15T06:11:53Z)
• Learning PAC-Bayes Priors for Probabilistic Neural Networks [32.01506699213665]
  Recent works have investigated deep learning models trained by optimising PAC-Bayes bounds, with priors that are learnt on subsets of the data. We ask what is the optimal amount of data which should be allocated for building the prior and show that the optimum may be dataset dependent.
  arXiv  Detail & Related papers  (2021-09-21T16:27:42Z)
• Learning and Certification under Instance-targeted Poisoning [49.55596073963654]
  We study PAC learnability and certification under instance-targeted poisoning attacks. We show that when the budget of the adversary scales sublinearly with the sample complexity, PAC learnability and certification are achievable. We empirically study the robustness of K nearest neighbour, logistic regression, multi-layer perceptron, and convolutional neural network on real data sets.
  arXiv  Detail & Related papers  (2021-05-18T17:48:15Z)
• RATT: Leveraging Unlabeled Data to Guarantee Generalization [96.08979093738024]
  We introduce a method that leverages unlabeled data to produce generalization bounds. We prove that our bound is valid for 0-1 empirical risk minimization. This work provides practitioners with an option for certifying the generalization of deep nets even when unseen labeled data is unavailable.
  arXiv  Detail & Related papers  (2021-05-01T17:05:29Z)
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data [48.4779912667317]
  Self-training algorithms have been very successful for learning with unlabeled data using neural networks. This work provides a unified theoretical analysis of self-training with deep networks for semi-supervised learning, unsupervised domain adaptation, and unsupervised learning.
  arXiv  Detail & Related papers  (2020-10-07T19:43:55Z)
• Tighter risk certificates for neural networks [10.462889461373226]
  We present two training objectives, used here for the first time in connection with training neural networks. We also re-implement a previously used training objective based on a classical PAC-Bayes bound. We compute risk certificates for the learnt predictors, based on part of the data used to learn the predictors.
  arXiv  Detail & Related papers  (2020-07-25T11:02:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.