Relaxation-Informed Training of Neural Network Surrogate Models

• URL: http://arxiv.org/abs/2604.22746v1
• Date: Fri, 24 Apr 2026 17:46:55 GMT
• Title: Relaxation-Informed Training of Neural Network Surrogate Models
• Abstract summary: ReLU neural networks trained surrogate models can be embedded exactly in mixed-integer linear programs (MILPs)<n>The tractability of the resulting MILP depends on structural properties of the network.<n>We show that the proposed regularizers can reduce MILP times by up to four orders of magnitude to an unregularized baseline.
• Score: 3.9722979176564763
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: ReLU neural networks trained as surrogate models can be embedded exactly in mixed-integer linear programs (MILPs), enabling global optimization over the learned function. The tractability of the resulting MILP depends on structural properties of the network, i.e., the number of binary variables in associated formulations and the tightness of the continuous LP relaxation. These properties are determined during training, yet standard training objectives (prediction loss with classical weight regularization) offer no mechanism to directly control them. This work studies training regularizers that directly target downstream MILP tractability. Specifically, we propose simple bound-based regularizers that penalize the big-M constants of MILP formulations and/or the number of unstable neurons. Moreover, we introduce an LP relaxation gap regularizer that explicitly penalizes the per-sample gap of the continuous relaxation at training points. We derive its associated gradient and provide an implementation from LP dual variables without custom automatic differentiation tools. We show that combining the above regularizers can approximate the full total derivative of the LP gap with respect to the network parameters, capturing both direct and indirect sensitivities. Experiments on non-convex benchmark functions and a two-stage stochastic programming problem with quantile neural network surrogates demonstrate that the proposed regularizers can reduce MILP solve times by up to four orders of magnitude relative to an unregularized baseline, while maintaining competitive surrogate model accuracy.

Related papers

• Randomized neural operator for parametric PDEs with fast training and conformal uncertainty quantification [47.720075875232226]
  We introduce PCA--RaNN, a neural operator that combines a PCA-based dimensionality reduction with fixed random features.<n>It recasts operator learning as fixed regression, reducing latent regression to three orders of magnitude.
  arXiv  Detail & Related papers  (2026-06-28T15:02:26Z)
• A Nonlinear Separation Principle: Applications to Neural Networks, Control and Learning [3.2241707757984788]
  This paper investigates continuous-time and discrete-time firing-rate and Hopfield recurrent neural networks (RNNs)<n>We introduce a nonlinear separation principle that guarantees global exponential stability for the interconnection of a contracting state-feedback controller and a contracting observer.<n>We derive sharp linear matrix inequality (LMI) conditions that guarantee the contractivity of both firing rate and Hopfield neural network architectures.
  arXiv  Detail & Related papers  (2026-04-16T17:12:32Z)
• FedSQ: Optimized Weight Averaging via Fixed Gating [1.2058208023553034]
  Federated learning enables collaborative training across organizations without sharing raw data.<n>We propose FedSQ (Federated Structural-Quantitative learning), a transfer-d neural federated procedure based on a DualCopy.<n>Fixing the gating reduces learning to within-regime affine refinements, which stabilizes aggregation under heterogeneous partitions.
  arXiv  Detail & Related papers  (2026-04-03T11:54:23Z)
• Recurrent Equivariant Constraint Modulation: Learning Per-Layer Symmetry Relaxation from Data [36.287199718605]
  Equivariant neural networks exploit underlying task symmetries to improve generalization.<n>We propose Recurrent Equivariant Constraint Modulation (RECM), a layer-wise constraint modulation mechanism.<n>RECM learns appropriate relaxation levels solely from the training signal and the symmetry properties of each layer's input-target distribution.
  arXiv  Detail & Related papers  (2026-02-02T21:59:35Z)
• A universal linearized subspace refinement framework for neural networks [1.636137854123538]
  We introduce Linearized Subspace Refinement (LSR), a general and architecture-agnostic subspace solver.<n>LSR yields a refined predictor with substantially improved accuracy over standard-trained solutions.<n>For problems with composite loss structures, we further introduce Iterative LSR, which alternates one-shot LSR with supervised nonlinear alignment.
  arXiv  Detail & Related papers  (2026-01-20T14:03:28Z)
• Mechanistic Insights into Grokking from the Embedding Layer [15.676058752772287]
  Grokking, a delayed generalization in neural networks, has been observed in Transformers and stagnates, but the components driving it remain underexplored.<n>We show that embeddings are central to grokking: introducing them intos induces delayed generalization in modular arithmetic tasks.<n>Our methods not only improve grokking dynamics but also extend to broader challenges in Transformer optimization, where bilinear interactions hinder efficient training.
  arXiv  Detail & Related papers  (2025-05-21T15:12:34Z)
• PseudoNeg-MAE: Self-Supervised Point Cloud Learning using Conditional Pseudo-Negative Embeddings [55.55445978692678]
  PseudoNeg-MAE enhances global feature representation of point cloud masked autoencoders by making them both discriminative and sensitive to transformations.<n>We propose a novel loss that explicitly penalizes invariant collapse, enabling the network to capture richer transformation cues while preserving discriminative representations.
  arXiv  Detail & Related papers  (2024-09-24T07:57:21Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Meta-Learning Adversarial Bandit Algorithms [55.72892209124227]
  We study online meta-learning with bandit feedback. We learn to tune online mirror descent generalization (OMD) with self-concordant barrier regularizers.
  arXiv  Detail & Related papers  (2023-07-05T13:52:10Z)
• Meta-Learning Adversarial Bandits [49.094361442409785]
  We study online learning with bandit feedback across multiple tasks, with the goal of improving average performance across tasks if they are similar according to some natural task-similarity measure. As the first to target the adversarial setting, we design a meta-algorithm that setting-specific guarantees for two important cases: multi-armed bandits (MAB) and bandit optimization (BLO) Our guarantees rely on proving that unregularized follow-the-leader combined with multiplicative weights is enough to online learn a non-smooth and non-B sequence.
  arXiv  Detail & Related papers  (2022-05-27T17:40:32Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.