Variational Distillation for Multi-View Learning
- URL: http://arxiv.org/abs/2206.09548v1
- Date: Mon, 20 Jun 2022 03:09:46 GMT
- Title: Variational Distillation for Multi-View Learning
- Authors: Xudong Tian, Zhizhong Zhang, Cong Wang, Wensheng Zhang, Yanyun Qu,
Lizhuang Ma, Zongze Wu, Yuan Xie, Dacheng Tao
- Abstract summary: We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning.
Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
- Score: 104.17551354374821
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Information Bottleneck (IB) based multi-view learning provides an information
theoretic principle for seeking shared information contained in heterogeneous
data descriptions. However, its great success is generally attributed to
estimate the multivariate mutual information which is intractable when the
network becomes complicated. Moreover, the representation learning tradeoff,
{\it i.e.}, prediction-compression and sufficiency-consistency tradeoff, makes
the IB hard to satisfy both requirements simultaneously. In this paper, we
design several variational information bottlenecks to exploit two key
characteristics ({\it i.e.}, sufficiency and consistency) for multi-view
representation learning. Specifically, we propose a Multi-View Variational
Distillation (MV$^2$D) strategy to provide a scalable, flexible and analytical
solution to fitting MI by giving arbitrary input of viewpoints but without
explicitly estimating it. Under rigorously theoretical guarantee, our approach
enables IB to grasp the intrinsic correlation between observations and semantic
labels, producing predictive and compact representations naturally. Also, our
information-theoretic constraint can effectively neutralize the sensitivity to
heterogeneous data by eliminating both task-irrelevant and view-specific
information, preventing both tradeoffs in multiple view cases. To verify our
theoretically grounded strategies, we apply our approaches to various
benchmarks under three different applications. Extensive experiments to
quantitatively and qualitatively demonstrate the effectiveness of our approach
against state-of-the-art methods.
Related papers
- An Information Criterion for Controlled Disentanglement of Multimodal Data [39.601584166020274]
Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities.
Disentangled Self-Supervised Learning (DisentangledSSL) is a novel self-supervised approach for learning disentangled representations.
arXiv Detail & Related papers (2024-10-31T14:57:31Z) - Differentiable Information Bottleneck for Deterministic Multi-view Clustering [9.723389925212567]
We propose a new differentiable information bottleneck (DIB) method, which provides a deterministic and analytical MVC solution.
Specifically, we first propose to directly fit the mutual information of high-dimensional spaces by leveraging normalized kernel Gram matrix.
Then, based on the new mutual information measurement, a deterministic multi-view neural network with analytical gradients is explicitly trained to parameterize IB principle.
arXiv Detail & Related papers (2024-03-23T02:13:22Z) - Constrained Multiview Representation for Self-supervised Contrastive
Learning [4.817827522417457]
We introduce a novel approach predicated on representation distance-based mutual information (MI) for measuring the significance of different views.
We harness multi-view representations extracted from the frequency domain, re-evaluating their significance based on mutual information.
arXiv Detail & Related papers (2024-02-05T19:09:33Z) - Triple Disentangled Representation Learning for Multimodal Affective Analysis [20.37986194570143]
Multimodal learning has exhibited a significant advantage in affective analysis tasks.
Many emerging studies focus on disentangling the modality-invariant and modality-specific representations from input data and then fusing them for prediction.
We propose a novel triple disentanglement approach, TriDiRA, which disentangles the modality-invariant, effective modality-specific and ineffective modality-specific representations from input data.
arXiv Detail & Related papers (2024-01-29T12:45:27Z) - Disentangled Representation Learning with Transmitted Information Bottleneck [57.22757813140418]
We present textbfDisTIB (textbfTransmitted textbfInformation textbfBottleneck for textbfDisd representation learning), a novel objective that navigates the balance between information compression and preservation.
arXiv Detail & Related papers (2023-11-03T03:18:40Z) - Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts
in Underspecified Visual Tasks [92.32670915472099]
We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs)
We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
arXiv Detail & Related papers (2023-10-03T17:37:52Z) - Which Mutual-Information Representation Learning Objectives are
Sufficient for Control? [80.2534918595143]
Mutual information provides an appealing formalism for learning representations of data.
This paper formalizes the sufficiency of a state representation for learning and representing the optimal policy.
Surprisingly, we find that two of these objectives can yield insufficient representations given mild and common assumptions on the structure of the MDP.
arXiv Detail & Related papers (2021-06-14T10:12:34Z) - Farewell to Mutual Information: Variational Distillation for Cross-Modal
Person Re-Identification [41.02729491273057]
The Information Bottleneck (IB) provides an information theoretic principle for representation learning.
We present a new strategy, Variational Self-Distillation (VSD), which provides a scalable, flexible and analytic solution.
We also introduce two other strategies, Variational Cross-Distillation (VCD) and Variational Mutual-Learning (VML)
arXiv Detail & Related papers (2021-04-07T02:19:41Z) - A Variational Information Bottleneck Approach to Multi-Omics Data
Integration [98.6475134630792]
We propose a deep variational information bottleneck (IB) approach for incomplete multi-view observations.
Our method applies the IB framework on marginal and joint representations of the observed views to focus on intra-view and inter-view interactions that are relevant for the target.
Experiments on real-world datasets show that our method consistently achieves gain from data integration and outperforms state-of-the-art benchmarks.
arXiv Detail & Related papers (2021-02-05T06:05:39Z) - Deep Partial Multi-View Learning [94.39367390062831]
We propose a novel framework termed Cross Partial Multi-View Networks (CPM-Nets)
We fifirst provide a formal defifinition of completeness and versatility for multi-view representation.
We then theoretically prove the versatility of the learned latent representations.
arXiv Detail & Related papers (2020-11-12T02:29:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.