FUNCK: Information Funnels and Bottlenecks for Invariant Representation
Learning
- URL: http://arxiv.org/abs/2211.01446v1
- Date: Wed, 2 Nov 2022 19:37:55 GMT
- Title: FUNCK: Information Funnels and Bottlenecks for Invariant Representation
Learning
- Authors: Jo\~ao Machado de Freitas and Bernhard C. Geiger
- Abstract summary: We investigate a set of related information funnels and bottleneck problems that claim to learn invariant representations from the data.
We propose a new element to this family of information-theoretic objectives: The Conditional Privacy Funnel with Side Information.
Given the generally intractable objectives, we derive tractable approximations using amortized variational inference parameterized by neural networks.
- Score: 7.804994311050265
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Learning invariant representations that remain useful for a downstream task
is still a key challenge in machine learning. We investigate a set of related
information funnels and bottleneck problems that claim to learn invariant
representations from the data. We also propose a new element to this family of
information-theoretic objectives: The Conditional Privacy Funnel with Side
Information, which we investigate in fully and semi-supervised settings. Given
the generally intractable objectives, we derive tractable approximations using
amortized variational inference parameterized by neural networks and study the
intrinsic trade-offs of these objectives. We describe empirically the proposed
approach and show that with a few labels it is possible to learn fair
classifiers and generate useful representations approximately invariant to
unwanted sources of variation. Furthermore, we provide insights about the
applicability of these methods in real-world scenarios with ordinary tabular
datasets when the data is scarce.
Related papers
- Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence [92.07601770031236]
We investigate semantically meaningful patterns in the attention heads of an encoder-only Transformer architecture.
We find that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization.
arXiv Detail & Related papers (2024-09-20T07:41:47Z) - Leveraging sparse and shared feature activations for disentangled
representation learning [112.22699167017471]
We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation.
We validate our approach on six real world distribution shift benchmarks, and different data modalities.
arXiv Detail & Related papers (2023-04-17T01:33:24Z) - Ignorance is Bliss: Robust Control via Information Gating [60.17644038829572]
Informational parsimony provides a useful inductive bias for learning representations that achieve better generalization by being robust to noise and spurious correlations.
We propose textitinformation gating as a way to learn parsimonious representations that identify the minimal information required for a task.
arXiv Detail & Related papers (2023-03-10T18:31:50Z) - RevUp: Revise and Update Information Bottleneck for Event Representation [16.54912614895861]
In machine learning, latent variables play a key role to capture the underlying structure of data, but they are often unsupervised.
We propose a semi-supervised information bottleneck-based model that enables the use of side knowledge to direct the learning of discrete latent variables.
We show that our approach generalizes an existing method of parameter injection, and perform an empirical case study of our approach on language-based event modeling.
arXiv Detail & Related papers (2022-05-24T17:54:59Z) - Information-Theoretic Odometry Learning [83.36195426897768]
We propose a unified information theoretic framework for learning-motivated methods aimed at odometry estimation.
The proposed framework provides an elegant tool for performance evaluation and understanding in information-theoretic language.
arXiv Detail & Related papers (2022-03-11T02:37:35Z) - Learning Bias-Invariant Representation by Cross-Sample Mutual
Information Minimization [77.8735802150511]
We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task.
The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator.
We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
arXiv Detail & Related papers (2021-08-11T21:17:02Z) - Robust Representation Learning via Perceptual Similarity Metrics [18.842322467828502]
Contrastive Input Morphing (CIM) is a representation learning framework that learns input-space transformations of the data.
We show that CIM is complementary to other mutual information-based representation learning techniques.
arXiv Detail & Related papers (2021-06-11T21:45:44Z) - Conditional Contrastive Learning: Removing Undesirable Information in
Self-Supervised Representations [108.29288034509305]
We develop conditional contrastive learning to remove undesirable information in self-supervised representations.
We demonstrate empirically that our methods can successfully learn self-supervised representations for downstream tasks.
arXiv Detail & Related papers (2021-06-05T10:51:26Z) - Nonlinear Invariant Risk Minimization: A Causal Approach [5.63479133344366]
We propose a learning paradigm that enables out-of-distribution generalization in the nonlinear setting.
We show identifiability of the data representation up to very simple transformations.
Extensive experiments on both synthetic and real-world datasets show that our approach significantly outperforms a variety of baseline methods.
arXiv Detail & Related papers (2021-02-24T15:38:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.