Expressing linear equality constraints in feedforward neural networks

• URL: http://arxiv.org/abs/2211.04395v1
• Date: Tue, 8 Nov 2022 17:39:05 GMT
• Title: Expressing linear equality constraints in feedforward neural networks
• Abstract summary: We introduce a new saddle-point Lagrangian with predictor auxiliary variables on which constraints are imposed. Elimination of the auxiliary variables leads to a dual minimization problem on the Lagrange multipliers introduced to satisfy the linear constraints. We obtain the surprising interpretation of Lagrange parameters as additional, penultimate layer hidden units with fixed weights stemming from the constraints.
• Score: 9.918927210224165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We seek to impose linear, equality constraints in feedforward neural networks. As top layer predictors are usually nonlinear, this is a difficult task if we seek to deploy standard convex optimization methods and strong duality. To overcome this, we introduce a new saddle-point Lagrangian with auxiliary predictor variables on which constraints are imposed. Elimination of the auxiliary variables leads to a dual minimization problem on the Lagrange multipliers introduced to satisfy the linear constraints. This minimization problem is combined with the standard learning problem on the weight matrices. From this theoretical line of development, we obtain the surprising interpretation of Lagrange parameters as additional, penultimate layer hidden units with fixed weights stemming from the constraints. Consequently, standard minimization approaches can be used despite the inclusion of Lagrange parameters -- a very satisfying, albeit unexpected, discovery. Examples ranging from multi-label classification to constrained autoencoders are envisaged in the future.

Related papers

• High-probability guarantees for linear accessibility in feature superposition [0.2538209532048867]
  Cross-feature interference constrains the linear accessibility of simultaneously active features.<n>We derive high-probability bounds for fixed supports under subgaussian noise.<n>These results provide a framework for evaluating sparse autoencoders, compositional generalization, and neural interpretability.
  arXiv  Detail & Related papers  (2026-09-09T00:26:35Z)
• Fitting Unknown Number of Hyperplanes with Manifold Optimization [57.48093263119306]
  Fitting unknown number of linearplanes to data is a fundamental yet challenging problem in machine learning.<n>Existing approaches often struggle with optimum optima or lack geometric consistency.
  arXiv  Detail & Related papers  (2026-05-27T14:02:20Z)
• Primal-Dual Guided Decoding for Constrained Discrete Diffusion [5.219025684249083]
  We propose primal-dual guided decoding, an inference-time method that formulates constrained generation as a KL-regularised optimisation problem.<n>At each denoising step, the method modifies token logits through an additive, constraint-dependent bias, with multipliers updated by mirror descent based on constraint violation.<n>We evaluate our approach on topical text generation, molecular design, and music playlist generation, showing that a single algorithm instantiated via domain-specific scoring functions improves constraint satisfaction while preserving relevant domain-specific quality metrics.
  arXiv  Detail & Related papers  (2026-05-10T20:55:19Z)
• Towards a Unified Analysis of Neural Networks in Nonparametric Instrumental Variable Regression: Optimization and Generalization [66.08522228989634]
  We establish the first global convergence result of neural networks for two stage least squares (2SLS) approach in nonparametric instrumental variable regression (NPIV)<n>This is achieved by adopting a lifted perspective through mean-field Langevin dynamics (MFLD)
  arXiv  Detail & Related papers  (2025-11-18T17:51:17Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation [0.0]
  contexts-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks.<n>Existing methods for neural networks suffer from numerical instabilities due to their reliance on classical formulas involving explicit Gram matrix computation and their subsequent inversion.<n>We propose a novel inversion-free regularized framework that is based entirely on stable decompositions and overcomes the numerical pitfalls of prior art.
  arXiv  Detail & Related papers  (2025-07-10T09:35:22Z)
• Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoders [6.178817969919849]
  We focus on two-layer linear denoising autoencoders trained under gradient flow.<n>A low-dimensional bottleneck layer that effectively enforces a rank constraint on the learned solution.<n> skip connection can mitigate the variance in denoising autoencoders.
  arXiv  Detail & Related papers  (2025-05-30T14:58:02Z)
• Scalable Approximate Algorithms for Optimal Transport Linear Models [0.769672852567215]
  We propose a novel framework for solving a general class of non-negative linear regression models with an entropy-regularized OT datafit term. We derive simple multiplicative updates for common penalty and datafit terms. This method is suitable for large-scale problems due to its simplicity of implementation and straightforward parallelization.
  arXiv  Detail & Related papers  (2025-04-06T20:37:25Z)
• A space-decoupling framework for optimization on bounded-rank matrices with orthogonally invariant constraints [4.917399520581689]
  We propose a space-decoupling framework for optimization on bounded-rank matrices. We show that the tangent cone of coupled constraints is the intersection of tangent cones of each constraint. We unveil the equivalence between the reformulated problem and the original problem.
  arXiv  Detail & Related papers  (2025-01-23T16:54:03Z)
• Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization [11.56128809794923]
  This paper explores a specific type of non sparsity-promoting regularization problems, namely those involving $ell_p-$ iterations of local property convergence.
  arXiv  Detail & Related papers  (2024-07-24T12:15:59Z)
• WANCO: Weak Adversarial Networks for Constrained Optimization problems [5.257895611010853]
  We first transform minimax problems into minimax problems using the augmented Lagrangian method. We then use two (or several) deep neural networks to represent the primal and dual variables respectively. The parameters in the neural networks are then trained by an adversarial process.
  arXiv  Detail & Related papers  (2024-07-04T05:37:48Z)
• Online Non-convex Optimization with Long-term Non-convex Constraints [7.68925638410622]
  A novel Follow-the-Perturbed-Leader type algorithm is proposed and analyzed for solving general long-term constrained optimization problems in an online manner.<n>The proposed is applied to tackle a long-term constrained source identification problem, validate the theoretical results and exhibit superior performance compared to existing methods.
  arXiv  Detail & Related papers  (2023-11-04T15:08:36Z)
• Network Topology Inference with Sparsity and Laplacian Constraints [18.447094648361453]
  We tackle the network topology by utilizing Laplacian constrained Gaussian graphical models. We show that an efficient projection algorithm is developed to solve the resulting problem.
  arXiv  Detail & Related papers  (2023-09-02T15:06:30Z)
• On Regularization and Inference with Label Constraints [62.60903248392479]
  We compare two strategies for encoding label constraints in a machine learning pipeline, regularization with constraints and constrained inference. For regularization, we show that it narrows the generalization gap by precluding models that are inconsistent with the constraints. For constrained inference, we show that it reduces the population risk by correcting a model's violation, and hence turns the violation into an advantage.
  arXiv  Detail & Related papers  (2023-07-08T03:39:22Z)
• Constrained Optimization via Exact Augmented Lagrangian and Randomized Iterative Sketching [55.28394191394675]
  We develop an adaptive inexact Newton method for equality-constrained nonlinear, nonIBS optimization problems. We demonstrate the superior performance of our method on benchmark nonlinear problems, constrained logistic regression with data from LVM, and a PDE-constrained problem.
  arXiv  Detail & Related papers  (2023-05-28T06:33:37Z)
• A Variational Inference Approach to Inverse Problems with Gamma Hyperpriors [60.489902135153415]
  This paper introduces a variational iterative alternating scheme for hierarchical inverse problems with gamma hyperpriors. The proposed variational inference approach yields accurate reconstruction, provides meaningful uncertainty quantification, and is easy to implement.
  arXiv  Detail & Related papers  (2021-11-26T06:33:29Z)
• A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learning [9.204659134755795]
  We consider the linear programming (LP) formulation for deep reinforcement learning. The augmented Lagrangian method suffers the double-sampling obstacle in solving the LP. A deep parameterized augment Lagrangian method is proposed.
  arXiv  Detail & Related papers  (2021-05-20T13:08:06Z)
• Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints [49.66841118264278]
  We introduce a series of challenging chaotic and extended-body systems to push the limits of current approaches. Our experiments show that Cartesian coordinates with explicit constraints lead to a 100x improvement in accuracy and data efficiency.
  arXiv  Detail & Related papers  (2020-10-26T13:35:16Z)
• Conditional gradient methods for stochastically constrained convex minimization [54.53786593679331]
  We propose two novel conditional gradient-based methods for solving structured convex optimization problems. The most important feature of our framework is that only a subset of the constraints is processed at each iteration. Our algorithms rely on variance reduction and smoothing used in conjunction with conditional gradient steps, and are accompanied by rigorous convergence guarantees.
  arXiv  Detail & Related papers  (2020-07-07T21:26:35Z)
• Competitive Mirror Descent [67.31015611281225]
  Constrained competitive optimization involves multiple agents trying to minimize conflicting objectives, subject to constraints. We propose competitive mirror descent (CMD): a general method for solving such problems based on first order information. As a special case we obtain a novel competitive multiplicative weights algorithm for problems on the positive cone.
  arXiv  Detail & Related papers  (2020-06-17T22:11:35Z)
• Lipschitz Bounds and Provably Robust Training by Laplacian Smoothing [7.4769019455423855]
  We formulate the adversarially robust learning problem as one of loss minimization with a Lipschitz constraint. We show that the saddle point of the associated Lagrangian is characterized by a Poisson equation with weighted Laplace operator. We design a provably robust training scheme using graph-based discretization of the input space and a primal-dual algorithm to converge to the Lagrangian's saddle point.
  arXiv  Detail & Related papers  (2020-06-05T22:02:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.