Efficient and provably convergent end-to-end training of deep neural networks with linear constraints

• URL: http://arxiv.org/abs/2605.11526v1
• Date: Tue, 12 May 2026 04:51:55 GMT
• Title: Efficient and provably convergent end-to-end training of deep neural networks with linear constraints
• Abstract summary: We introduce an efficiently computable HS-Jacobian to the projection layer.<n>We prove that the HS-Jacobian is a conservative mapping for the projection operator onto the polyhedral set.<n>We establish convergence guarantees of the HS-Jacobian based Adam algorithm for training linearly constrained deep neural networks.
• Score: 12.559930224041544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training a deep neural network with the outputs of selected layers satisfying linear constraints is required in many contemporary data-driven applications. While this can be achieved by incorporating projection layers into the neural network, its end-to-end training remains challenging due to the lack of rigorous theory and efficient algorithms for backpropagation. A key difficulty in developing the theory and efficient algorithms for backpropagation arose from the nonsmoothness of the solution mapping of the projection layer. To address this bottleneck, we introduce an efficiently computable HS-Jacobian to the projection layer. Importantly, we prove that the HS-Jacobian is a conservative mapping for the projection operator onto the polyhedral set, enabling its seamless integration into the nonsmooth automatic differentiation framework for backpropagation. Therefore, many efficient algorithms, such as Adam, can be applied for end-to-end training of deep neural networks with linear constraints. Particularly, we establish convergence guarantees of the HS-Jacobian based Adam algorithm for training linearly constrained deep neural networks. Extensive experiment results on several important applications, including finance, computer vision, and network architecture design, demonstrate the superior performance of our method compared to other existing popular methods.

Related papers

• Self-Abstraction Learning for Effective and Stable Training of Deep Neural Networks [1.787702109751332]
  Training large-scale deep neural networks effectively and stably is essential for applying deep learning across various fields.<n>To overcome these limitations, we introduce Self-Abstraction Learning (SAL), a hierarchical framework.<n>In SAL, networks are arranged by structural complexity, where the simplest topmost network is trained first and its hidden and output layers serve as guidance for the successively more complex networks below.
  arXiv  Detail & Related papers  (2026-04-27T11:04:18Z)
• Coordinate Encoding on Linear Grids for Physics-Informed Neural Networks [0.12277343096128708]
  We propose a PINN-based method equipped with a coordinate-encoding layer on linear grid cells.<n>The proposed method improves the training convergence speed by separating the local domains using grid cells.<n>It reduces the overall computational cost by using axis-independent linear grid cells.
  arXiv  Detail & Related papers  (2026-03-24T01:45:46Z)
• Concurrent Training and Layer Pruning of Deep Neural Networks [0.0]
  We propose an algorithm capable of identifying and eliminating irrelevant layers of a neural network during the early stages of training. We employ a structure using residual connections around nonlinear network sections that allow the flow of information through the network once a nonlinear section is pruned.
  arXiv  Detail & Related papers  (2024-06-06T23:19:57Z)
• Robust Stochastically-Descending Unrolled Networks [85.6993263983062]
  Deep unrolling is an emerging learning-to-optimize method that unrolls a truncated iterative algorithm in the layers of a trainable neural network.<n>We show that convergence guarantees and generalizability of the unrolled networks are still open theoretical problems.<n>We numerically assess unrolled architectures trained under the proposed constraints in two different applications.
  arXiv  Detail & Related papers  (2023-12-25T18:51:23Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Losses [0.0]
  We propose an integrated method that combines the strengths of both FFA and shallow backpropagation. We show that training neural networks with the Integrated Forward-Forward Algorithm has the potential of generating neural networks with advantageous features like robustness.
  arXiv  Detail & Related papers  (2023-05-22T12:10:47Z)
• The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
  We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF. Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples. In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
  arXiv  Detail & Related papers  (2023-03-17T02:01:11Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Analytically Tractable Inference in Deep Neural Networks [0.0]
  Tractable Approximate Inference (TAGI) algorithm was shown to be a viable and scalable alternative to backpropagation for shallow fully-connected neural networks. We are demonstrating how TAGI matches or exceeds the performance of backpropagation, for training classic deep neural network architectures.
  arXiv  Detail & Related papers  (2021-03-09T14:51:34Z)
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks [50.42141893913188]
  We study a distributed variable for large-scale AUC for a neural network as with a deep neural network. Our model requires a much less number of communication rounds and still a number of communication rounds in theory. Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
  arXiv  Detail & Related papers  (2020-05-05T18:08:23Z)
• MSE-Optimal Neural Network Initialization via Layer Fusion [68.72356718879428]
  Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. The use of gradient combined nonvolutionity renders learning susceptible to novel problems. We propose fusing neighboring layers of deeper networks that are trained with random variables.
  arXiv  Detail & Related papers  (2020-01-28T18:25:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.