Disentanglement Analysis with Partial Information Decomposition
- URL: http://arxiv.org/abs/2108.13753v1
- Date: Tue, 31 Aug 2021 11:09:40 GMT
- Title: Disentanglement Analysis with Partial Information Decomposition
- Authors: Seiya Tokui, Issei Sato
- Abstract summary: disentangled representations aim at reversing the process by mapping data to multiple random variables that individually capture distinct generative factors.
Current disentanglement metrics are designed to measure the concentration, e.g., absolute deviation, variance, or entropy, of each variable conditioned by each generative factor.
In this work, we use the Partial Information Decomposition framework to evaluate information sharing between more than two variables, and build a framework, including a new disentanglement metric.
- Score: 31.56299813238937
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Given data generated from multiple factors of variation that cooperatively
transform their appearance, disentangled representations aim at reversing the
process by mapping data to multiple random variables that individually capture
distinct generative factors. As the concept is intuitive but abstract, one
needs to quantify it with disentanglement metrics to evaluate and compare the
quality of disentangled representations between different models. Current
disentanglement metrics are designed to measure the concentration, e.g.,
absolute deviation, variance, or entropy, of each variable conditioned by each
generative factor, optionally offset by the concentration of its marginal
distribution, and compare it among different variables. When representations
consist of more than two variables, such metrics may fail to detect the
interplay between them as they only measure pairwise interactions. In this
work, we use the Partial Information Decomposition framework to evaluate
information sharing between more than two variables, and build a framework,
including a new disentanglement metric, for analyzing how the representations
encode the generative factors distinctly, redundantly, and cooperatively. We
establish an experimental protocol to assess how each metric evaluates
increasingly entangled representations and confirm through artificial and
realistic settings that the proposed metric correctly responds to entanglement.
Our results are expected to promote information theoretic understanding of
disentanglement and lead to further development of metrics as well as learning
methods.
Related papers
- Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data.
Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
arXiv Detail & Related papers (2024-04-24T09:04:36Z) - Enriching Disentanglement: From Logical Definitions to Quantitative Metrics [59.12308034729482]
Disentangling the explanatory factors in complex data is a promising approach for data-efficient representation learning.
We establish relationships between logical definitions and quantitative metrics to derive theoretically grounded disentanglement metrics.
We empirically demonstrate the effectiveness of the proposed metrics by isolating different aspects of disentangled representations.
arXiv Detail & Related papers (2023-05-19T08:22:23Z) - The Counterfactual-Shapley Value: Attributing Change in System Metrics [10.804568364995982]
A key component of an attribution question is estimating counterfactual: the (hypothetical) change in the system metric due to a specified change in a single input.
We propose a method to estimate counterfactuals using time-series predictive models and construct an attribution score, CF-Shapley.
As a real-world application, we analyze a query-ad matching system with the goal of attributing observed change in a metric for ad matching density.
arXiv Detail & Related papers (2022-08-17T16:48:20Z) - On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
Generalization captures a model's ability to classify unseen data.
Invariance measures consistency of model predictions on transformations of the data.
From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
arXiv Detail & Related papers (2022-07-14T17:08:25Z) - Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
We study how to discover interpretable equivariances from data.
Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes.
Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
arXiv Detail & Related papers (2022-04-07T17:59:19Z) - Learning Disentangled Representations with Latent Variation
Predictability [102.4163768995288]
This paper defines the variation predictability of latent disentangled representations.
Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs.
We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
arXiv Detail & Related papers (2020-07-25T08:54:26Z) - TCMI: a non-parametric mutual-dependence estimator for multivariate
continuous distributions [0.0]
Total cumulative mutual information (TCMI) is a measure of the relevance of mutual dependences.
TCMI is a non-parametric, robust, and deterministic measure that facilitates comparisons and rankings between feature sets.
arXiv Detail & Related papers (2020-01-30T08:42:25Z) - Disentangled Representation Learning with Wasserstein Total Correlation [90.44329632061076]
We introduce Wasserstein total correlation in both variational autoencoder and Wasserstein autoencoder settings to learn disentangled latent representations.
A critic is adversarially trained along with the main objective to estimate the Wasserstein total correlation term.
We show that the proposed approach has comparable performances on disentanglement with smaller sacrifices in reconstruction abilities.
arXiv Detail & Related papers (2019-12-30T05:31:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.