Causal Graphs Underlying Generative Models: Path to Learning with
Limited Data
- URL: http://arxiv.org/abs/2207.07174v1
- Date: Thu, 14 Jul 2022 19:20:30 GMT
- Title: Causal Graphs Underlying Generative Models: Path to Learning with
Limited Data
- Authors: Samuel C. Hoffman, Kahini Wadhawan, Payel Das, Prasanna Sattigeri,
Karthikeyan Shanmugam
- Abstract summary: We use perturbation experiments on latent codes of a pre-trained generative autoencoder to uncover a causal graph that is implied by the generative model.
We show that one can fit an effective causal graph that models a structural equation model between latent codes.
Using a pre-trained RNN-based generative autoencoder trained on a dataset of peptide sequences, we demonstrate that the learnt causal graph can be used to predict a specific property for sequences which are unseen.
- Score: 45.38541020492725
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Training generative models that capture rich semantics of the data and
interpreting the latent representations encoded by such models are very
important problems in unsupervised learning. In this work, we provide a simple
algorithm that relies on perturbation experiments on latent codes of a
pre-trained generative autoencoder to uncover a causal graph that is implied by
the generative model. We leverage pre-trained attribute classifiers and perform
perturbation experiments to check for influence of a given latent variable on a
subset of attributes. Given this, we show that one can fit an effective causal
graph that models a structural equation model between latent codes taken as
exogenous variables and attributes taken as observed variables. One interesting
aspect is that a single latent variable controls multiple overlapping subsets
of attributes unlike conventional approach that tries to impose full
independence. Using a pre-trained RNN-based generative autoencoder trained on a
dataset of peptide sequences, we demonstrate that the learnt causal graph from
our algorithm between various attributes and latent codes can be used to
predict a specific property for sequences which are unseen. We compare
prediction models trained on either all available attributes or only the ones
in the Markov blanket and empirically show that in both the unsupervised and
supervised regimes, typically, using the predictor that relies on Markov
blanket attributes generalizes better for out-of-distribution sequences.
Related papers
- Exploring Beyond Logits: Hierarchical Dynamic Labeling Based on Embeddings for Semi-Supervised Classification [49.09505771145326]
We propose a Hierarchical Dynamic Labeling (HDL) algorithm that does not depend on model predictions and utilizes image embeddings to generate sample labels.
Our approach has the potential to change the paradigm of pseudo-label generation in semi-supervised learning.
arXiv Detail & Related papers (2024-04-26T06:00:27Z) - Robust Graph Representation Learning via Predictive Coding [46.22695915912123]
Predictive coding is a message-passing framework initially developed to model information processing in the brain.
In this work, we build models that rely on the message-passing rule of predictive coding.
We show that the proposed models are comparable to standard ones in terms of performance in both inductive and transductive tasks.
arXiv Detail & Related papers (2022-12-09T03:58:22Z) - Mutual Exclusivity Training and Primitive Augmentation to Induce
Compositionality [84.94877848357896]
Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models.
We analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias and the tendency to memorize whole examples.
We show substantial empirical improvements using standard sequence-to-sequence models on two widely-used compositionality datasets.
arXiv Detail & Related papers (2022-11-28T17:36:41Z) - Learning Sparse Latent Representations for Generator Model [7.467412443287767]
We present a new unsupervised learning method to enforce sparsity on the latent space for the generator model.
Our model consists of only one top-down generator network that maps the latent variable to the observed data.
arXiv Detail & Related papers (2022-09-20T18:58:24Z) - Discovering Invariant Rationales for Graph Neural Networks [104.61908788639052]
Intrinsic interpretability of graph neural networks (GNNs) is to find a small subset of the input graph's features.
We propose a new strategy of discovering invariant rationale (DIR) to construct intrinsically interpretable GNNs.
arXiv Detail & Related papers (2022-01-30T16:43:40Z) - Learning Consistent Deep Generative Models from Sparse Data via
Prediction Constraints [16.48824312904122]
We develop a new framework for learning variational autoencoders and other deep generative models.
We show that these two contributions -- prediction constraints and consistency constraints -- lead to promising image classification performance.
arXiv Detail & Related papers (2020-12-12T04:18:50Z) - Autoencoding Variational Autoencoder [56.05008520271406]
We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency.
We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
arXiv Detail & Related papers (2020-12-07T14:16:14Z) - Discriminative, Generative and Self-Supervised Approaches for
Target-Agnostic Learning [8.666667951130892]
generative and self-supervised learning models are shown to perform well at the task.
Our derived theorem for the pseudo-likelihood theory also shows that they are related for inferring a joint distribution model.
arXiv Detail & Related papers (2020-11-12T15:03:40Z) - Unsupervised Controllable Generation with Self-Training [90.04287577605723]
controllable generation with GANs remains a challenging research problem.
We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training.
Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
arXiv Detail & Related papers (2020-07-17T21:50:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.