Scalable and Efficient Methods for Uncertainty Estimation and Reduction in Deep Learning

• URL: http://arxiv.org/abs/2401.07145v1
• Date: Sat, 13 Jan 2024 19:30:34 GMT
• Title: Scalable and Efficient Methods for Uncertainty Estimation and Reduction in Deep Learning
• Authors: Soyed Tuhin Ahmed
• Abstract summary: This paper explores scalable and efficient methods for uncertainty estimation and reduction in deep learning. We tackle the inherent uncertainties arising from out-of-distribution inputs and hardware non-idealities. Our approach encompasses problem-aware training algorithms, novel NN topologies, and hardware co-design solutions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Neural networks (NNs) can achieved high performance in various fields such as computer vision, and natural language processing. However, deploying NNs in resource-constrained safety-critical systems has challenges due to uncertainty in the prediction caused by out-of-distribution data, and hardware non-idealities. To address the challenges of deploying NNs in resource-constrained safety-critical systems, this paper summarizes the (4th year) PhD thesis work that explores scalable and efficient methods for uncertainty estimation and reduction in deep learning, with a focus on Computation-in-Memory (CIM) using emerging resistive non-volatile memories. We tackle the inherent uncertainties arising from out-of-distribution inputs and hardware non-idealities, crucial in maintaining functional safety in automated decision-making systems. Our approach encompasses problem-aware training algorithms, novel NN topologies, and hardware co-design solutions, including dropout-based \emph{binary} Bayesian Neural Networks leveraging spintronic devices and variational inference techniques. These innovations significantly enhance OOD data detection, inference accuracy, and energy efficiency, thereby contributing to the reliability and robustness of NN implementations.

Related papers

• Scaling #DNN-Verification Tools with Efficient Bound Propagation and Parallel Computing [57.49021927832259]
  Deep Neural Networks (DNNs) are powerful tools that have shown extraordinary results in many scenarios. However, their intricate designs and lack of transparency raise safety concerns when applied in real-world applications. Formal Verification (FV) of DNNs has emerged as a valuable solution to provide provable guarantees on the safety aspect.
  arXiv  Detail & Related papers  (2023-12-10T13:51:25Z)
• Scale-Dropout: Estimating Uncertainty in Deep Neural Networks Using Stochastic Scale [0.7025445595542577]
  Uncertainty estimation in Neural Networks (NNs) is vital in improving reliability and confidence in predictions, particularly in safety-critical applications. BayNNs with Dropout as an approximation offer a systematic approach to uncertainty, but they inherently suffer from high hardware overhead in terms of power, memory, and quantifying. We introduce a novel Spintronic memory-based CIM architecture for the proposed BayNN that achieves more than $100times$ energy savings compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-11-27T13:41:20Z)
• Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networks [23.7125322065694]
  Uncertainty quantification (UQ) is important for reliability assessment and enhancement of machine learning models. We create statistically guaranteed schemes to principally emphcharacterize, and emphremove, the uncertainty of over- parameterized neural networks. In particular, our approach, based on what we call a procedural-noise-correcting (PNC) predictor, removes the procedural uncertainty by using only emphone auxiliary network that is trained on a suitably labeled dataset.
  arXiv  Detail & Related papers  (2023-06-09T05:15:53Z)
• APPRAISER: DNN Fault Resilience Analysis Employing Approximation Errors [1.1091582432763736]
  Deep Neural Networks (DNNs) in safety-critical applications raise new reliability concerns. State-of-the-art methods for fault injection by emulation incur a spectrum of time-, design- and control-complexity problems. APPRAISER is proposed that applies functional approximation for a non-conventional purpose and employs approximate computing errors.
  arXiv  Detail & Related papers  (2023-05-31T10:53:46Z)
• Physics-Informed Neural Networks for Time-Domain Simulations: Accuracy, Computational Cost, and Flexibility [0.0]
  Physics-Informed Neural Networks (PINNs) have emerged as a promising solution for drastically accelerating computations of non-linear dynamical systems. This work investigates the applicability of these methods for power system dynamics, focusing on the dynamic response to load disturbances. To facilitate a deeper understanding, this paper also present a new regularisation of Neural Network (NN) training by introducing a gradient-based term in the loss function.
  arXiv  Detail & Related papers  (2023-03-15T23:53:32Z)
• On Robust Numerical Solver for ODE via Self-Attention Mechanism [82.95493796476767]
  We explore training efficient and robust AI-enhanced numerical solvers with a small data size by mitigating intrinsic noise disturbances. We first analyze the ability of the self-attention mechanism to regulate noise in supervised learning and then propose a simple-yet-effective numerical solver, Attr, which introduces an additive self-attention mechanism to the numerical solution of differential equations.
  arXiv  Detail & Related papers  (2023-02-05T01:39:21Z)
• Backward Reachability Analysis of Neural Feedback Loops: Techniques for Linear and Nonlinear Systems [59.57462129637796]
  This paper presents a backward reachability approach for safety verification of closed-loop systems with neural networks (NNs) The presence of NNs in the feedback loop presents a unique set of problems due to the nonlinearities in their activation functions and because NN models are generally not invertible. We present frameworks for calculating BP over-approximations for both linear and nonlinear systems with control policies represented by feedforward NNs.
  arXiv  Detail & Related papers  (2022-09-28T13:17:28Z)
• The Unreasonable Effectiveness of Deep Evidential Regression [72.30888739450343]
  A new approach with uncertainty-aware regression-based neural networks (NNs) shows promise over traditional deterministic methods and typical Bayesian NNs. We detail the theoretical shortcomings and analyze the performance on synthetic and real-world data sets, showing that Deep Evidential Regression is a quantification rather than an exact uncertainty.
  arXiv  Detail & Related papers  (2022-05-20T10:10:32Z)
• Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework [13.573645522781712]
  Deep neural networks (DNNs) and spiking neural networks (SNNs) offer state-of-the-art results on resource-constrained edge devices. These systems are required to maintain correct functionality under diverse security and reliability threats. This paper first discusses existing approaches to address energy efficiency, reliability, and security issues at different system layers.
  arXiv  Detail & Related papers  (2021-09-20T20:22:56Z)
• Learning to Solve the AC-OPF using Sensitivity-Informed Deep Neural Networks [52.32646357164739]
  We propose a deep neural network (DNN) to solve the solutions of the optimal power flow (ACOPF) The proposed SIDNN is compatible with a broad range of OPF schemes. It can be seamlessly integrated in other learning-to-OPF schemes.
  arXiv  Detail & Related papers  (2021-03-27T00:45:23Z)
• Increasing the Confidence of Deep Neural Networks by Coverage Analysis [71.57324258813674]
  This paper presents a lightweight monitoring architecture based on coverage paradigms to enhance the model against different unsafe inputs. Experimental results show that the proposed approach is effective in detecting both powerful adversarial examples and out-of-distribution inputs.
  arXiv  Detail & Related papers  (2021-01-28T16:38:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.