A lift for input-convex neural network training

• URL: http://arxiv.org/abs/2605.24274v1
• Date: Fri, 22 May 2026 22:59:11 GMT
• Title: A lift for input-convex neural network training
• Abstract summary: In Inputdimensional neural networks (ICNNs) are used for logconcave density estimation, convexpotential normalizing flows, optimal transport, and transportmap inversion for posteriors.<n>The standard projected gradient descent (PGD) onto the non-negative cone applies a hard, nonsmooth projection.<n>The differentiable alternative, softplus reparametrization, attenuates the gradient exponentially in the weight magnitude, stalling training with dead interlayer weights loss.<n>We show that the lift reaches a lower test loss than both PGD and direct softplus, and turns a plateau-bounded training into a
• Score: 3.142113135607563
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Input-convex neural networks (ICNNs) are widely used for log-concave density estimation, convex-potential normalizing flows, optimal transport, and transport-map inversion for high-dimensional Bayesian posteriors. These tasks share a structural constraint: the inter-layer weights of the ICNN must remain non-negative. The standard recipe, projected gradient descent (PGD) onto the non-negative cone, applies a hard, non-smooth projection -- the stiff-penalty limit of an ADMM-style constraint splitting -- and its classical convergence guarantees do not transfer to the non-smooth ICNN training landscape; the differentiable alternative, softplus reparametrization, attenuates the gradient exponentially in the weight magnitude, stalling training with dead inter-layer weights and plateaued loss. Inspired by parameter-extension lifts of PDE-constrained inverse problems, we propose the lift: instead of constraining the inter-layer weights directly, we train an unconstrained hypernetwork that emits them from a permutation-invariant summary of the input batch. This adds stochasticity to the training dynamics that softens the loss landscape, letting the iterates escape the gradient-attenuated region where direct softplus stalls. We trace this softening to three structural ingredients -- a learnable bias acting as slack, a hypernetwork body that conditions on the target batch, and a cross-covariance coupling the two through batch stochasticity -- and prove each one necessary: deleting any single ingredient collapses the cross-covariance that carries the softening. On log-concave energy-based modeling from one-dimensional toy targets to image-flavored latents, and convex-potential normalizing flows on a 21-dimensional tabular benchmark, we show that the lift reaches a lower test loss than both PGD and direct softplus, and turns a plateau-bounded training trajectory into a valley-descending one.

Related papers

• Feed-Forward Steering in Transformer Residual Dynamics [4.109803241412197]
  We extend attention-only Transformer residual directions by incorporating the feed-forward network (FFN) term as a local steering field on each token state.<n>The resulting theory predicts that the tangential component of the FFN field is necessary for motion in residual-direction space.<n>As a practical application, layers with small commutator defects can be approximately parallelized with only a modest increase in loss.
  arXiv  Detail & Related papers  (2026-08-03T11:15:31Z)
• Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks [1.1516147824168732]
  Training binary neural networks (BNNs) from scratch is dominated by the straight-through estimator (STE)<n>We introduce StoMPP, which gradually replaces clipped weights and activations with their hard binary counterparts layer by layer from input to output.<n>As a standalone training rule, it provides a fully STE-free procedure that improves over vanilla STE with gains that grow with depth.
  arXiv  Detail & Related papers  (2026-06-26T06:37:57Z)
• When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks [22.81761236732655]
  We study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.<n>Our analysis tracks the trajectory through three phases: an alignment phase in which hidden weights progressively align with the planted direction while the output weights maintain the correct sign pattern.<n>We establish novel uniform concentration results that hold along the entire trajectory, and are essential for obtaining order-wise optimal sample complexity.
  arXiv  Detail & Related papers  (2026-06-03T05:44:30Z)
• Out-of-distribution transfer of PDE foundation models to material dynamics under extreme loading [86.6550968435969]
  Most PDE foundation models are pretrained and fine-tuned on fluid-centric benchmarks.<n>We benchmark out-of-distribution transfer on two discontinuity-dominated regimes in which shocks, evolving interfaces, and fracture produce highly non-smooth fields.<n>We evaluate two open-source PDE foundation models, POSEIDON and MORPH, and compare fine-tuning from pretrained weights against training from scratch across training-set sizes to quantify sample efficiency under distribution shift.
  arXiv  Detail & Related papers  (2026-03-04T18:19:35Z)
• Data-Free PINNs for Compressible Flows: Mitigating Spectral Bias and Gradient Pathologies via Mach-Guided Scaling and Hybrid Convolutions [0.0]
  This paper presents a fully data-free Physics-Informed Neural Network (PINN) capable of solving compressible inviscid flows.<n>The proposed framework successfully captures the detached bow shock without referential data.
  arXiv  Detail & Related papers  (2026-03-01T09:04:18Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Dual-Balancing for Physics-Informed Neural Networks [5.8096456298528745]
  Physics-informed neural networks (PINNs) have emerged as a new learning paradigm for solving partial differential equations (PDEs)<n> PINNs still suffer from poor accuracy and slow convergence due to the intractable multi-objective optimization issue.<n>We propose a novel Dual-Balanced PINN (DB-PINN), which dynamically adjusts loss weights by integrating inter-balancing and intra-balancing.
  arXiv  Detail & Related papers  (2025-05-16T11:00:54Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Learning Discretized Neural Networks under Ricci Flow [48.47315844022283]
  We study Discretized Neural Networks (DNNs) composed of low-precision weights and activations.<n>DNNs suffer from either infinite or zero gradients due to the non-differentiable discrete function during training.
  arXiv  Detail & Related papers  (2023-02-07T10:51:53Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations [52.493315075385325]
  We show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with homogeneous activation functions. We propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network.
  arXiv  Detail & Related papers  (2020-08-07T02:55:28Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.