Deep Generative Models with Hard Linear Equality Constraints

• URL: http://arxiv.org/abs/2502.05416v2
• Date: Wed, 12 Feb 2025 08:58:06 GMT
• Title: Deep Generative Models with Hard Linear Equality Constraints
• Authors: Ruoyan Li, Dipti Ranjan Sahu, Guy Van den Broeck, Zhe Zeng,
• Abstract summary: We propose a probabilistically sound approach for enforcing the hard constraints into DGMs to generate constraint-compliant data. We carry out experiments with various DGM model architectures over five image datasets and three scientific applications. Ours not only guarantees the satisfaction of constraints in generation but also archives superior generative performance than the other methods across every benchmark.
• Score: 24.93865980946986
• License:
• Abstract: While deep generative models~(DGMs) have demonstrated remarkable success in capturing complex data distributions, they consistently fail to learn constraints that encode domain knowledge and thus require constraint integration. Existing solutions to this challenge have primarily relied on heuristic methods and often ignore the underlying data distribution, harming the generative performance. In this work, we propose a probabilistically sound approach for enforcing the hard constraints into DGMs to generate constraint-compliant and realistic data. This is achieved by our proposed gradient estimators that allow the constrained distribution, the data distribution conditioned on constraints, to be differentiably learned. We carry out extensive experiments with various DGM model architectures over five image datasets and three scientific applications in which domain knowledge is governed by linear equality constraints. We validate that the standard DGMs almost surely generate data violating the constraints. Among all the constraint integration strategies, ours not only guarantees the satisfaction of constraints in generation but also archives superior generative performance than the other methods across every benchmark.

Related papers

• Hard Constraint Guided Flow Matching for Gradient-Free Generation of PDE Solutions [41.558608119074755]
  We introduce a novel framework for adapting pre-trained, unconstrained flow-matching models to satisfy constraints exactly in a zero-shot manner without requiring expensive computations or fine-tuning. Our framework, ECI sampling, alternates between extrapolation (E), correction (C) and Generative (I) stages to ensure accurate integration of constraint information while preserving the validity of the generation. We demonstrate the effectiveness of our approach across various PDE systems, showing that ECI-guided generation strictly adheres to physical constraints and accurately captures complex distribution shifts induced by these constraints.
  arXiv  Detail & Related papers  (2024-12-02T18:36:26Z)
• Guaranteed Generation from Large Language Models [28.157857382660563]
  Large language models (LLMs) are increasingly used across various applications. We propose GUARD, a simple yet effective approach that combines an autoregressive proposal distribution with rejection sampling. These experiments show that GUARD achieves perfect constraint satisfaction while almost preserving the ideal distribution with highly improved inference efficiency.
  arXiv  Detail & Related papers  (2024-10-09T09:39:55Z)
• Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL [57.202733701029594]
  We propose Decision Mamba, a novel multi-grained state space model (SSM) with a self-evolving policy learning strategy. To address these challenges, we propose Decision Mamba, a novel multi-grained state space model (SSM) with a self-evolving policy learning strategy. To mitigate the overfitting issue on noisy trajectories, a self-evolving policy is proposed by using progressive regularization.
  arXiv  Detail & Related papers  (2024-06-08T10:12:00Z)
• How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data [57.97035325253996]
  We show how Constrained Deep Generative Models (C-DGMs) can be transformed into realistic synthetic data models. C-DGMs are able to exploit the background knowledge expressed by the constraints to outperform their standard counterparts.
  arXiv  Detail & Related papers  (2024-02-07T13:22:05Z)
• Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
  Federated learning aims to train models across different clients without the sharing of data for privacy considerations. We study how data heterogeneity affects the representations of the globally aggregated models. We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
  arXiv  Detail & Related papers  (2022-10-01T09:04:17Z)
• Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization [23.302060306322506]
  Real-world data often has multiple distribution shifts over different attributes. No state-of-the-art DG algorithm performs consistently well on all shifts. We develop Causally Adaptive Constraint Minimization (CACM), an algorithm that uses knowledge about the data-generating process to adaptively identify and apply the correct independence constraints for regularization.
  arXiv  Detail & Related papers  (2022-06-15T22:35:06Z)
• Toward Certified Robustness Against Real-World Distribution Shifts [65.66374339500025]
  We train a generative model to learn perturbations from data and define specifications with respect to the output of the learned model. A unique challenge arising from this setting is that existing verifiers cannot tightly approximate sigmoid activations. We propose a general meta-algorithm for handling sigmoid activations which leverages classical notions of counter-example-guided abstraction refinement.
  arXiv  Detail & Related papers  (2022-06-08T04:09:13Z)
• Principled Knowledge Extrapolation with GANs [92.62635018136476]
  We study counterfactual synthesis from a new perspective of knowledge extrapolation. We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem. Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
  arXiv  Detail & Related papers  (2022-05-21T08:39:42Z)
• SaDe: Learning Models that Provably Satisfy Domain Constraints [16.46852109556965]
  We present a machine learning approach that can handle a wide variety of constraints, and guarantee that these constraints will be satisfied by the model even on unseen data. We cast machine learning as a maximum satisfiability problem, and solve it using a novel algorithm SaDe which combines constraint satisfaction with gradient descent.
  arXiv  Detail & Related papers  (2021-12-01T15:18:03Z)
• An Integer Linear Programming Framework for Mining Constraints from Data [81.60135973848125]
  We present a general framework for mining constraints from data. In particular, we consider the inference in structured output prediction as an integer linear programming (ILP) problem. We show that our approach can learn to solve 9x9 Sudoku puzzles and minimal spanning tree problems from examples without providing the underlying rules.
  arXiv  Detail & Related papers  (2020-06-18T20:09:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.