Differentiable Projection for Constrained Deep Learning

• URL: http://arxiv.org/abs/2111.10785v1
• Date: Sun, 21 Nov 2021 10:32:43 GMT
• Title: Differentiable Projection for Constrained Deep Learning
• Authors: Dou Huang, Haoran Zhang, Xuan Song and Ryosuke Shibasaki
• Abstract summary: In some applications, some prior knowledge could be easily obtained, such as constraints which the ground truth observation follows. In this paper, we propose to use a differentiable projection layer in DNN instead of directly solving time-consuming KKT conditions. The proposed projection method is differentiable, and no heavy computation is required.
• Score: 17.228410662469994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) have achieved extraordinary performance in solving different tasks in various fields. However, the conventional DNN model is steadily approaching the ground-truth value through loss backpropagation. In some applications, some prior knowledge could be easily obtained, such as constraints which the ground truth observation follows. Here, we try to give a general approach to incorporate information from these constraints to enhance the performance of the DNNs. Theoretically, we could formulate these kinds of problems as constrained optimization problems that KKT conditions could solve. In this paper, we propose to use a differentiable projection layer in DNN instead of directly solving time-consuming KKT conditions. The proposed projection method is differentiable, and no heavy computation is required. Finally, we also conducted some experiments using a randomly generated synthetic dataset and image segmentation task using the PASCAL VOC dataset to evaluate the performance of the proposed projection method. Experimental results show that the projection method is sufficient and outperforms baseline methods.

Related papers

• Reliable Projection Based Unsupervised Learning for Semi-Definite QCQP with Application of Beamforming Optimization [11.385703484113552]
  In this paper, we investigate a special class of quadratic (QCQP) with semi-definite constraints. We propose a neural network (NN) as a promising method to obtain a high-performing constraint solution. Unsupervised learning is used, so the NN can be effectively efficiently without labels.
  arXiv  Detail & Related papers  (2024-07-04T06:26:01Z)
• Best-Effort Adaptation [62.00856290846247]
  We present a new theoretical analysis of sample reweighting methods, including bounds holding uniformly over the weights. We show how these bounds can guide the design of learning algorithms that we discuss in detail. We report the results of a series of experiments demonstrating the effectiveness of our best-effort adaptation and domain adaptation algorithms.
  arXiv  Detail & Related papers  (2023-05-10T00:09:07Z)
• Variational Linearized Laplace Approximation for Bayesian Deep Learning [11.22428369342346]
  We propose a new method for approximating Linearized Laplace Approximation (LLA) using a variational sparse Gaussian Process (GP) Our method is based on the dual RKHS formulation of GPs and retains, as the predictive mean, the output of the original DNN. It allows for efficient optimization, which results in sub-linear training time in the size of the training dataset.
  arXiv  Detail & Related papers  (2023-02-24T10:32:30Z)
• Validation Diagnostics for SBI algorithms based on Normalizing Flows [55.41644538483948]
  This work proposes easy to interpret validation diagnostics for multi-dimensional conditional (posterior) density estimators based on NF. It also offers theoretical guarantees based on results of local consistency. This work should help the design of better specified models or drive the development of novel SBI-algorithms.
  arXiv  Detail & Related papers  (2022-11-17T15:48:06Z)
• Exploiting Temporal Structures of Cyclostationary Signals for Data-Driven Single-Channel Source Separation [98.95383921866096]
  We study the problem of single-channel source separation (SCSS) We focus on cyclostationary signals, which are particularly suitable in a variety of application domains. We propose a deep learning approach using a U-Net architecture, which is competitive with the minimum MSE estimator.
  arXiv  Detail & Related papers  (2022-08-22T14:04:56Z)
• Data-informed Deep Optimization [3.331457049134526]
  We propose a data-informed deep optimization (DiDo) approach to solve high-dimensional design problems. We use a deep neural network (DNN) to learn the feasible region and to sample feasible points for fitting the objective function. Our results indicate that the DiDo approach empowered by DNN is flexible and promising for solving general high-dimensional design problems in practice.
  arXiv  Detail & Related papers  (2021-07-17T02:53:54Z)
• Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation [101.22379613810881]
  We consider data-driven optimization problems where one must maximize a function given only queries at a fixed set of points. This problem setting emerges in many domains where function evaluation is a complex and expensive process. We propose a tractable approximation that allows us to scale our method to high-capacity neural network models.
  arXiv  Detail & Related papers  (2021-02-16T06:04:27Z)
• High Dimensional Level Set Estimation with Bayesian Neural Network [58.684954492439424]
  This paper proposes novel methods to solve the high dimensional Level Set Estimation problems using Bayesian Neural Networks. For each problem, we derive the corresponding theoretic information based acquisition function to sample the data points. Numerical experiments on both synthetic and real-world datasets show that our proposed method can achieve better results compared to existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-12-17T23:21:53Z)
• Quantifying Statistical Significance of Neural Network-based Image Segmentation by Selective Inference [23.97765106673937]
  We use a conditional selective inference (SI) framework to compute exact (non-asymptotic) valid p-values for the segmentation results. Our proposed method can successfully control the false positive rate, has good performance in terms of computational efficiency, and provides good results when applied to medical image data.
  arXiv  Detail & Related papers  (2020-10-05T07:16:40Z)
• Differentiable Causal Discovery from Interventional Data [141.41931444927184]
  We propose a theoretically-grounded method based on neural networks that can leverage interventional data. We show that our approach compares favorably to the state of the art in a variety of settings.
  arXiv  Detail & Related papers  (2020-07-03T15:19:17Z)
• Exponentially improved detection and correction of errors in experimental systems using neural networks [0.0]
  We introduce the use of two machine learning algorithms to create an empirical model of an experimental apparatus. This is able to reduce the number of measurements necessary for generic optimisation tasks exponentially. We demonstrate both algorithms at the example of detecting and compensating stray electric fields in an ion trap.
  arXiv  Detail & Related papers  (2020-05-18T22:42:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.