Constraining Generative Models for Engineering Design with Negative Data

• URL: http://arxiv.org/abs/2306.15166v2
• Date: Mon, 02 Dec 2024 14:20:12 GMT
• Title: Constraining Generative Models for Engineering Design with Negative Data
• Abstract summary: We introduce a novel training method to guide a generative model toward constraint-satisfying outputs.<n>Our negative-data generative model (NDGM) formulation easily outperforms classic models.
• Score: 11.432911164773488
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative models have recently achieved remarkable success and widespread adoption in society, yet they often struggle to generate realistic and accurate outputs. This challenge extends beyond language and vision into fields like engineering design, where safety-critical engineering standards and non-negotiable physical laws tightly constrain what outputs are considered acceptable. In this work, we introduce a novel training method to guide a generative model toward constraint-satisfying outputs using `negative data' -- examples of what to avoid. Our negative-data generative model (NDGM) formulation easily outperforms classic models, generating 1/6 as many constraint-violating samples using 1/8 as much data in certain problems. It also consistently outperforms other baselines, achieving a balance between constraint satisfaction and distributional similarity that is unsurpassed by any other model in 12 of the 14 problems tested. This widespread superiority is rigorously demonstrated across numerous synthetic tests and real engineering problems, such as ship hull synthesis with hydrodynamic constraints and vehicle design with impact safety constraints. Our benchmarks showcase both the best-in-class performance of our new NDGM formulation and the overall dominance of NDGMs versus classic generative models. We publicly release the code and benchmarks at https://github.com/Lyleregenwetter/NDGMs.

Related papers

• QueST: Incentivizing LLMs to Generate Difficult Problems [77.75835742350644]
  Large Language Models have achieved strong performance on reasoning tasks, solving competition-level coding and math problems.<n>Existing competitive coding datasets contain only thousands to tens of thousands of problems.<n>We propose QueST, a novel framework which combines difficulty-aware graph sampling and difficulty-aware rejection fine-tuning.
  arXiv  Detail & Related papers  (2025-10-20T16:29:53Z)
• Catch-Only-One: Non-Transferable Examples for Model-Specific Authorization [26.668781698446832]
  Recent AI regulations call for data that remain useful for innovation while resistant to misuse.<n>We propose non-transferable examples (NEs), a training-free and data-agnostic input-side usage-control mechanism.<n>We establish formal bounds that guarantee utility for the authorized model and quantify deviation for unauthorized ones.
  arXiv  Detail & Related papers  (2025-10-13T03:43:11Z)
• Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models [28.127179660781838]
  Existing world models for autonomous driving struggle with long-horizon generation and generalization to challenging scenarios.<n>We develop a model using simple design choices, and without additional supervision or sensors, such as maps, depth, or multiple cameras.<n>We show that our model yields state-of-the-art performance, despite having only 469M parameters and being trained on 280h of video data.
  arXiv  Detail & Related papers  (2025-07-17T14:29:34Z)
• Masked Conditioning for Deep Generative Models [0.0]
  We introduce a novel masked-conditioning approach that enables generative models to work with sparse, mixed-type data.<n>We show that small models trained on limited data can be coupled with large pretrained foundation models to improve generation quality.
  arXiv  Detail & Related papers  (2025-05-22T14:33:03Z)
• Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>One core challenge of evaluation in the large language model (LLM) era is the generalization issue.<n>We propose Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• Deep Generative Models with Hard Linear Equality Constraints [24.93865980946986]
  We propose a probabilistically sound approach for enforcing the hard constraints into DGMs to generate constraint-compliant data. We carry out experiments with various DGM model architectures over five image datasets and three scientific applications. Ours not only guarantees the satisfaction of constraints in generation but also archives superior generative performance than the other methods across every benchmark.
  arXiv  Detail & Related papers  (2025-02-08T02:53:32Z)
• Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens [53.99177152562075]
  Scaling up autoregressive models in vision has not proven as beneficial as in large language models. We focus on two critical factors: whether models use discrete or continuous tokens, and whether tokens are generated in a random or fixed order using BERT- or GPT-like transformer architectures. Our results show that while all models scale effectively in terms of validation loss, their evaluation performance -- measured by FID, GenEval score, and visual quality -- follows different trends.
  arXiv  Detail & Related papers  (2024-10-17T17:59:59Z)
• Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines [74.42485647685272]
  We focus on Generative Masked Language Models (GMLMs) We train a model to fit conditional probabilities of the data distribution via masking, which are subsequently used as inputs to a Markov Chain to draw samples from the model. We adapt the T5 model for iteratively-refined parallel decoding, achieving 2-3x speedup in machine translation with minimal sacrifice in quality.
  arXiv  Detail & Related papers  (2024-07-22T18:00:00Z)
• Leveraging Latent Diffusion Models for Training-Free In-Distribution Data Augmentation for Surface Defect Detection [9.784793380119806]
  We introduce DIAG, a training-free Diffusion-based In-distribution Anomaly Generation pipeline for data augmentation. Unlike conventional image generation techniques, we implement a human-in-the-loop pipeline, where domain experts provide multimodal guidance to the model. We demonstrate the efficacy and versatility of DIAG with respect to state-of-the-art data augmentation approaches on the challenging KSDD2 dataset.
  arXiv  Detail & Related papers  (2024-07-04T14:28:52Z)
• SynthForge: Synthesizing High-Quality Face Dataset with Controllable 3D Generative Models [8.004597666699036]
  Recent advancements in generative models have unlocked the capabilities to render photo-realistic data in a controllable fashion. However, using the data generated using such models for training downstream tasks remains under-explored, mainly due to the lack of 3D consistent annotations. Our experiments show competitive performance against state-of-the-art models using only generated synthetic data, demonstrating potential for solving downstream tasks.
  arXiv  Detail & Related papers  (2024-06-12T03:15:15Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data [57.97035325253996]
  We show how Constrained Deep Generative Models (C-DGMs) can be transformed into realistic synthetic data models. C-DGMs are able to exploit the background knowledge expressed by the constraints to outperform their standard counterparts.
  arXiv  Detail & Related papers  (2024-02-07T13:22:05Z)
• Three-Stage Adjusted Regression Forecasting (TSARF) for Software Defect Prediction [5.826476252191368]
  Nonhomogeneous Poisson process (NHPP) SRGM are the most commonly employed models. Increased model complexity presents a challenge in identifying robust and computationally efficient algorithms.
  arXiv  Detail & Related papers  (2024-01-31T02:19:35Z)
• Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule [55.08778142798106]
  We propose auxiliary tasks that exploit the alignment between the original and corrected sentences. We formulate each task as a sequence-to-sequence problem and perform multi-task training. We find that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance.
  arXiv  Detail & Related papers  (2023-11-20T14:50:12Z)
• AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectories [18.266786462036553]
  We propose an effective DD framework named AST, standing for Alignment with Smooth and high-quality expert Trajectories. We conduct extensive experiments on datasets of different scales, sizes, and resolutions.
  arXiv  Detail & Related papers  (2023-10-16T16:13:53Z)
• Self-Supervised Dataset Distillation for Transfer Learning [77.4714995131992]
  We propose a novel problem of distilling an unlabeled dataset into a set of small synthetic samples for efficient self-supervised learning (SSL) We first prove that a gradient of synthetic samples with respect to a SSL objective in naive bilevel optimization is textitbiased due to randomness originating from data augmentations or masking. We empirically validate the effectiveness of our method on various applications involving transfer learning.
  arXiv  Detail & Related papers  (2023-10-10T10:48:52Z)
• Robust Monocular Depth Estimation under Challenging Conditions [81.57697198031975]
  State-of-the-art monocular depth estimation approaches are highly unreliable under challenging illumination and weather conditions. We tackle these safety-critical issues with md4all: a simple and effective solution that works reliably under both adverse and ideal conditions.
  arXiv  Detail & Related papers  (2023-08-18T17:59:01Z)
• Non-Invasive Fairness in Learning through the Lens of Data Drift [88.37640805363317]
  We show how to improve the fairness of Machine Learning models without altering the data or the learning algorithm. We use a simple but key insight: the divergence of trends between different populations, and, consecutively, between a learned model and minority populations, is analogous to data drift. We explore two strategies (model-splitting and reweighing) to resolve this drift, aiming to improve the overall conformance of models to the underlying data.
  arXiv  Detail & Related papers  (2023-03-30T17:30:42Z)
• How to train your draGAN: A task oriented solution to imbalanced classification [15.893327571516016]
  This paper proposes a unique, performance-oriented, data-generating strategy that utilizes a new architecture, coined draGAN. The samples are generated with the objective of optimizing the classification model's performance, rather than similarity to the real data. Empirically we show the superiority of draGAN, but also highlight some of its shortcomings.
  arXiv  Detail & Related papers  (2022-11-18T07:37:34Z)
• Simultaneous Improvement of ML Model Fairness and Performance by Identifying Bias in Data [1.76179873429447]
  We propose a data preprocessing technique that can detect instances ascribing a specific kind of bias that should be removed from the dataset before training. In particular, we claim that in the problem settings where instances exist with similar feature but different labels caused by variation in protected attributes, an inherent bias gets induced in the dataset.
  arXiv  Detail & Related papers  (2022-10-24T13:04:07Z)
• ClusterQ: Semantic Feature Distribution Alignment for Data-Free Quantization [111.12063632743013]
  We propose a new and effective data-free quantization method termed ClusterQ. To obtain high inter-class separability of semantic features, we cluster and align the feature distribution statistics. We also incorporate the intra-class variance to solve class-wise mode collapse.
  arXiv  Detail & Related papers  (2022-04-30T06:58:56Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning [25.85044477227461]
  Models that are more accurate on the out-of-distribution data relative to this baseline exhibit "effective robustness" We find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.
  arXiv  Detail & Related papers  (2021-06-30T06:21:42Z)
• Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
  We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-18T15:13:00Z)
• Convex Smoothed Autoencoder-Optimal Transport model [0.0]
  We develop a new generative model capable of generating samples which resemble the observed data, and is free from mode collapse and mode mixture. Our model is inspired by the recently proposed Autoencoder- Optimal Transport (AE-OT) model and tries to improve on it by addressing the problems faced by the AE-OT model itself.
  arXiv  Detail & Related papers  (2021-01-14T15:55:20Z)
• Learning Consistent Deep Generative Models from Sparse Data via Prediction Constraints [16.48824312904122]
  We develop a new framework for learning variational autoencoders and other deep generative models. We show that these two contributions -- prediction constraints and consistency constraints -- lead to promising image classification performance.
  arXiv  Detail & Related papers  (2020-12-12T04:18:50Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.