Expressing linear equality constraints in feedforward neural networks

• URL: http://arxiv.org/abs/2211.04395v1
• Date: Tue, 8 Nov 2022 17:39:05 GMT
• Title: Expressing linear equality constraints in feedforward neural networks
• Authors: Anand Rangarajan, Pan He, Jaemoon Lee, Tania Banerjee, Sanjay Ranka
• Abstract summary: We introduce a new saddle-point Lagrangian with predictor auxiliary variables on which constraints are imposed. Elimination of the auxiliary variables leads to a dual minimization problem on the Lagrange multipliers introduced to satisfy the linear constraints. We obtain the surprising interpretation of Lagrange parameters as additional, penultimate layer hidden units with fixed weights stemming from the constraints.
• Score: 9.918927210224165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We seek to impose linear, equality constraints in feedforward neural networks. As top layer predictors are usually nonlinear, this is a difficult task if we seek to deploy standard convex optimization methods and strong duality. To overcome this, we introduce a new saddle-point Lagrangian with auxiliary predictor variables on which constraints are imposed. Elimination of the auxiliary variables leads to a dual minimization problem on the Lagrange multipliers introduced to satisfy the linear constraints. This minimization problem is combined with the standard learning problem on the weight matrices. From this theoretical line of development, we obtain the surprising interpretation of Lagrange parameters as additional, penultimate layer hidden units with fixed weights stemming from the constraints. Consequently, standard minimization approaches can be used despite the inclusion of Lagrange parameters -- a very satisfying, albeit unexpected, discovery. Examples ranging from multi-label classification to constrained autoencoders are envisaged in the future.

Related papers

• COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation [0.0]
  contexts-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks.<n>Existing methods for neural networks suffer from numerical instabilities due to their reliance on classical formulas involving explicit Gram matrix computation and their subsequent inversion.<n>We propose a novel inversion-free regularized framework that is based entirely on stable decompositions and overcomes the numerical pitfalls of prior art.
  arXiv  Detail & Related papers  (2025-07-10T09:35:22Z)
• Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoders [6.178817969919849]
  We focus on two-layer linear denoising autoencoders trained under gradient flow.<n>A low-dimensional bottleneck layer that effectively enforces a rank constraint on the learned solution.<n> skip connection can mitigate the variance in denoising autoencoders.
  arXiv  Detail & Related papers  (2025-05-30T14:58:02Z)
• Scalable Approximate Algorithms for Optimal Transport Linear Models [0.769672852567215]
  We propose a novel framework for solving a general class of non-negative linear regression models with an entropy-regularized OT datafit term. We derive simple multiplicative updates for common penalty and datafit terms. This method is suitable for large-scale problems due to its simplicity of implementation and straightforward parallelization.
  arXiv  Detail & Related papers  (2025-04-06T20:37:25Z)
• A space-decoupling framework for optimization on bounded-rank matrices with orthogonally invariant constraints [4.917399520581689]
  We propose a space-decoupling framework for optimization on bounded-rank matrices. We show that the tangent cone of coupled constraints is the intersection of tangent cones of each constraint. We unveil the equivalence between the reformulated problem and the original problem.
  arXiv  Detail & Related papers  (2025-01-23T16:54:03Z)
• Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization [11.56128809794923]
  This paper explores a specific type of non sparsity-promoting regularization problems, namely those involving $ell_p-$ iterations of local property convergence.
  arXiv  Detail & Related papers  (2024-07-24T12:15:59Z)
• WANCO: Weak Adversarial Networks for Constrained Optimization problems [5.257895611010853]
  We first transform minimax problems into minimax problems using the augmented Lagrangian method. We then use two (or several) deep neural networks to represent the primal and dual variables respectively. The parameters in the neural networks are then trained by an adversarial process.
  arXiv  Detail & Related papers  (2024-07-04T05:37:48Z)
• Network Topology Inference with Sparsity and Laplacian Constraints [18.447094648361453]
  We tackle the network topology by utilizing Laplacian constrained Gaussian graphical models. We show that an efficient projection algorithm is developed to solve the resulting problem.
  arXiv  Detail & Related papers  (2023-09-02T15:06:30Z)
• On Regularization and Inference with Label Constraints [62.60903248392479]
  We compare two strategies for encoding label constraints in a machine learning pipeline, regularization with constraints and constrained inference. For regularization, we show that it narrows the generalization gap by precluding models that are inconsistent with the constraints. For constrained inference, we show that it reduces the population risk by correcting a model's violation, and hence turns the violation into an advantage.
  arXiv  Detail & Related papers  (2023-07-08T03:39:22Z)
• Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketching [55.28394191394675]
  We develop an adaptive inexact Newton method for equality-constrained nonlinear, nonIBS optimization problems. We demonstrate the superior performance of our method on benchmark nonlinear problems, constrained logistic regression with data from LVM, and a PDE-constrained problem.
  arXiv  Detail & Related papers  (2023-05-28T06:33:37Z)
• A Variational Inference Approach to Inverse Problems with Gamma Hyperpriors [60.489902135153415]
  This paper introduces a variational iterative alternating scheme for hierarchical inverse problems with gamma hyperpriors. The proposed variational inference approach yields accurate reconstruction, provides meaningful uncertainty quantification, and is easy to implement.
  arXiv  Detail & Related papers  (2021-11-26T06:33:29Z)
• A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learning [9.204659134755795]
  We consider the linear programming (LP) formulation for deep reinforcement learning. The augmented Lagrangian method suffers the double-sampling obstacle in solving the LP. A deep parameterized augment Lagrangian method is proposed.
  arXiv  Detail & Related papers  (2021-05-20T13:08:06Z)
• Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints [49.66841118264278]
  We introduce a series of challenging chaotic and extended-body systems to push the limits of current approaches. Our experiments show that Cartesian coordinates with explicit constraints lead to a 100x improvement in accuracy and data efficiency.
  arXiv  Detail & Related papers  (2020-10-26T13:35:16Z)
• Conditional gradient methods for stochastically constrained convex minimization [54.53786593679331]
  We propose two novel conditional gradient-based methods for solving structured convex optimization problems. The most important feature of our framework is that only a subset of the constraints is processed at each iteration. Our algorithms rely on variance reduction and smoothing used in conjunction with conditional gradient steps, and are accompanied by rigorous convergence guarantees.
  arXiv  Detail & Related papers  (2020-07-07T21:26:35Z)
• Competitive Mirror Descent [67.31015611281225]
  Constrained competitive optimization involves multiple agents trying to minimize conflicting objectives, subject to constraints. We propose competitive mirror descent (CMD): a general method for solving such problems based on first order information. As a special case we obtain a novel competitive multiplicative weights algorithm for problems on the positive cone.
  arXiv  Detail & Related papers  (2020-06-17T22:11:35Z)
• Lipschitz Bounds and Provably Robust Training by Laplacian Smoothing [7.4769019455423855]
  We formulate the adversarially robust learning problem as one of loss minimization with a Lipschitz constraint. We show that the saddle point of the associated Lagrangian is characterized by a Poisson equation with weighted Laplace operator. We design a provably robust training scheme using graph-based discretization of the input space and a primal-dual algorithm to converge to the Lagrangian's saddle point.
  arXiv  Detail & Related papers  (2020-06-05T22:02:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.