Compressed Predictive Information Coding
- URL: http://arxiv.org/abs/2203.02051v1
- Date: Thu, 3 Mar 2022 22:47:58 GMT
- Title: Compressed Predictive Information Coding
- Authors: Rui Meng, Tianyi Luo, Kristofer Bouchard
- Abstract summary: We develop a novel information-theoretic framework, Compressed Predictive Information Coding (CPIC), to extract useful representations from dynamic data.
We derive variational bounds of the CPIC loss which induces the latent space to capture information that is maximally predictive.
We demonstrate that CPIC is able to recover the latent space of noisy dynamical systems with low signal-to-noise ratios.
- Score: 6.220929746808418
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Unsupervised learning plays an important role in many fields, such as
artificial intelligence, machine learning, and neuroscience. Compared to static
data, methods for extracting low-dimensional structure for dynamic data are
lagging. We developed a novel information-theoretic framework, Compressed
Predictive Information Coding (CPIC), to extract useful representations from
dynamic data. CPIC selectively projects the past (input) into a linear subspace
that is predictive about the compressed data projected from the future
(output). The key insight of our framework is to learn representations by
minimizing the compression complexity and maximizing the predictive information
in latent space. We derive variational bounds of the CPIC loss which induces
the latent space to capture information that is maximally predictive. Our
variational bounds are tractable by leveraging bounds of mutual information. We
find that introducing stochasticity in the encoder robustly contributes to
better representation. Furthermore, variational approaches perform better in
mutual information estimation compared with estimates under a Gaussian
assumption. We demonstrate that CPIC is able to recover the latent space of
noisy dynamical systems with low signal-to-noise ratios, and extracts features
predictive of exogenous variables in neuroscience data.
Related papers
Err
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.