Diffusion Model-Augmented Behavioral Cloning

• URL: http://arxiv.org/abs/2302.13335v4
• Date: Mon, 3 Jun 2024 16:17:28 GMT
• Title: Diffusion Model-Augmented Behavioral Cloning
• Abstract summary: This work proposes an imitation learning framework that benefits from modeling both the conditional and joint probability of the expert distribution. Our proposed Diffusion Model-Augmented Behavioral Cloning (DBC) employs a diffusion model trained to model expert behaviors and learns a policy to optimize both the BC loss (conditional) and our proposed diffusion model loss (joint)
• Score: 4.363232795241617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Imitation learning addresses the challenge of learning by observing an expert's demonstrations without access to reward signals from environments. Most existing imitation learning methods that do not require interacting with environments either model the expert distribution as the conditional probability p(a|s) (e.g., behavioral cloning, BC) or the joint probability p(s, a). Despite the simplicity of modeling the conditional probability with BC, it usually struggles with generalization. While modeling the joint probability can improve generalization performance, the inference procedure is often time-consuming, and the model can suffer from manifold overfitting. This work proposes an imitation learning framework that benefits from modeling both the conditional and joint probability of the expert distribution. Our proposed Diffusion Model-Augmented Behavioral Cloning (DBC) employs a diffusion model trained to model expert behaviors and learns a policy to optimize both the BC loss (conditional) and our proposed diffusion model loss (joint). DBC outperforms baselines in various continuous control tasks in navigation, robot arm manipulation, dexterous manipulation, and locomotion. We design additional experiments to verify the limitations of modeling either the conditional probability or the joint probability of the expert distribution, as well as compare different generative models. Ablation studies justify the effectiveness of our design choices.

Related papers

• Multi-Objective Learning for Diffusion Models: A Statistical Theory under Semi-Supervised Learning [83.65494573724904]
  We develop a principled MOL framework for diffusion models with limited data.<n>We propose a two-stage training procedure that first fits lightweight specialist models from limited paired data.<n>We establish bounds generalization showing that the required number of paired samples only depends on the complexity of the specialist model classes.
  arXiv  Detail & Related papers  (2026-05-24T18:19:40Z)
• A Probabilistic Consensus-Driven Approach for Robust Counterfactual Explanations [2.4788097726838667]
  We propose a novel approach that jointly models the data distribution and the space of plausible model decisions to ensure robustness to model changes.<n>Our method effectively pushes CFEs toward regions that are both plausible and stable across model changes.
  arXiv  Detail & Related papers  (2026-04-19T15:31:18Z)
• Generalized Interpolating Discrete Diffusion [65.74168524007484]
  Masked diffusion is a popular choice due to its simplicity and effectiveness. We derive the theoretical backbone of a family of general interpolating discrete diffusion processes. Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise.
  arXiv  Detail & Related papers  (2025-03-06T14:30:55Z)
• Continuous Bayesian Model Selection for Multivariate Causal Discovery [22.945274948173182]
  Current causal discovery approaches require restrictive model assumptions or assume access to interventional data to ensure structure identifiability. Recent work has shown that Bayesian model selection can greatly improve accuracy by exchanging restrictive modelling for more flexible assumptions. We demonstrate the competitiveness of our approach on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-11-15T12:55:05Z)
• InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion [53.90516061351706]
  We present InterHandGen, a novel framework that learns the generative prior of two-hand interaction. For sampling, we combine anti-penetration and synthesis-free guidance to enable plausible generation. Our method significantly outperforms baseline generative models in terms of plausibility and diversity.
  arXiv  Detail & Related papers  (2024-03-26T06:35:55Z)
• A Probabilistic Fluctuation based Membership Inference Attack for Diffusion Models [32.15773300068426]
  Membership Inference Attack (MIA) identifies whether a record exists in a machine learning model's training set by querying the model. We propose a Probabilistic Fluctuation Assessing Membership Inference Attack (PFAMI) PFAMI can improve the attack success rate (ASR) by about 27.9% when compared with the best baseline.
  arXiv  Detail & Related papers  (2023-08-23T14:00:58Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• On the Generalization of Diffusion Model [42.447639515467934]
  We define the generalization of the generative model, which is measured by the mutual information between the generated data and the training set. We show that for the empirical optimal diffusion model, the data generated by a deterministic sampler are all highly related to the training set, thus poor generalization. We propose another training objective whose empirical optimal solution has no potential generalization problem.
  arXiv  Detail & Related papers  (2023-05-24T04:27:57Z)
• Bi-Noising Diffusion: Towards Conditional Diffusion Models with Generative Restoration Priors [64.24948495708337]
  We introduce a new method that brings predicted samples to the training data manifold using a pretrained unconditional diffusion model. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution, colorization, turbulence removal, and image-deraining tasks.
  arXiv  Detail & Related papers  (2022-12-14T17:26:35Z)
• Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose? [0.2836066255205732]
  We contribute to micro-data model-based reinforcement learning (MBRL) by rigorously comparing popular generative models. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin. We also found that deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counterparts.
  arXiv  Detail & Related papers  (2021-07-24T11:38:25Z)
• PSD Representations for Effective Probability Models [117.35298398434628]
  We show that a recently proposed class of positive semi-definite (PSD) models for non-negative functions is particularly suited to this end. We characterize both approximation and generalization capabilities of PSD models, showing that they enjoy strong theoretical guarantees. Our results open the way to applications of PSD models to density estimation, decision theory and inference.
  arXiv  Detail & Related papers  (2021-06-30T15:13:39Z)
• A Twin Neural Model for Uplift [59.38563723706796]
  Uplift is a particular case of conditional treatment effect modeling. We propose a new loss function defined by leveraging a connection with the Bayesian interpretation of the relative risk. We show our proposed method is competitive with the state-of-the-art in simulation setting and on real data from large scale randomized experiments.
  arXiv  Detail & Related papers  (2021-05-11T16:02:39Z)
• Generalization and Memorization: The Bias Potential Model [9.975163460952045]
  generative models and density estimators behave quite differently from models for learning functions. For the bias potential model, we show that dimension-independent generalization accuracy is achievable if early stopping is adopted. In the long term, the model either memorizes the samples or diverges.
  arXiv  Detail & Related papers  (2020-11-29T04:04:54Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.