Compression phase is not necessary for generalization in representation learning

• URL: http://arxiv.org/abs/2102.07402v1
• Date: Mon, 15 Feb 2021 09:02:45 GMT
• Title: Compression phase is not necessary for generalization in representation learning
• Abstract summary: It is debatable if the compression phase is necessary to generalize the input-output relations extracted from training data. We investigated this through experiments with various species of autoencoders and evaluated their information processing phase. We conclude that the compression phase is not necessary for generalization in representation learning.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The outstanding performance of deep learning in various fields has been a fundamental query, which can be potentially examined using information theory that interprets the learning process as the transmission and compression of information. Information plane analyses of the mutual information between the input-hidden-output layers demonstrated two distinct learning phases of fitting and compression. It is debatable if the compression phase is necessary to generalize the input-output relations extracted from training data. In this study, we investigated this through experiments with various species of autoencoders and evaluated their information processing phase with an accurate kernel-based estimator of mutual information. Given sufficient training data, vanilla autoencoders demonstrated the compression phase, which was amplified after imposing sparsity regularization for hidden activities. However, we found that the compression phase is not universally observed in different species of autoencoders, including variational autoencoders, that have special constraints on network weights or manifold of hidden space. These types of autoencoders exhibited perfect generalization ability for test data without requiring the compression phase. Thus, we conclude that the compression phase is not necessary for generalization in representation learning.

Related papers

• Tensor network compression using fluid dynamics as a testbed: Analytical foundations in one dimension [0.0]
  High performance computers produce extreme-scale data sets that require sampling or compression if they are to be used to their full potential.<n>Existing data compression techniques typically exploit features such as sparsity in the data, homogeneity in the data, or it a priori knowledge of what subsets of data are of most interest.<n> Fluid dynamics data in general do not exhibit these features and so are attractive test beds for generic compression techniques that are objective, robust, and tuneable with respect to information lost due to compression.
  arXiv  Detail & Related papers  (2026-06-04T11:12:08Z)
• Seq2Seq2Seq: Lossless Data Compression via Discrete Latent Transformers and Reinforcement Learning [3.2641459166493405]
  We propose a novel compression method based on Reinforcement Learning applied to a T5 language model architecture.<n>This approach enables the compression of data into sequences of tokens rather than traditional vector representations.<n>By leveraging the latent information within language models, our system effectively compresses data without requiring explicit content understanding.
  arXiv  Detail & Related papers  (2026-02-12T16:30:55Z)
• Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model [20.1054266241262]
  We investigate how data distribution impacts compression quality, including two dimensions: input data and intrinsic data.<n>We show that encoder-measured input entropy negatively correlates with compression quality, while decoder-measured entropy shows no significant relationship under a frozen-decoder setting.
  arXiv  Detail & Related papers  (2026-02-02T08:01:57Z)
• InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression [114.03378443007074]
  Current tokenizers rigidly compress all content at a fixed rate, leading to redundancy or information loss.<n>This paper introduces InfoTok, a principled framework for adaptive video tokenization.<n>We develop a transformer-based adaptive compressor that enables adaptive tokenization.
  arXiv  Detail & Related papers  (2025-12-18T17:13:59Z)
• Compressive Meta-Learning [49.300635370079874]
  Compressive learning is a framework that enables efficient processing by using random, non-linear features.<n>We propose a framework that meta-learns both the encoding and decoding stages of compressive learning methods.<n>We explore multiple applications -- including neural network-based compressive PCA, compressive ridge regression, compressive k-means, and autoencoders.
  arXiv  Detail & Related papers  (2025-08-14T22:08:06Z)
• Neural Normalized Compression Distance and the Disconnect Between Compression and Classification [42.98054061480037]
  We develop a Neural NCD and compare LLMs to classic general-purpose algorithms like gzip. We find that classification accuracy is not predictable by compression rate alone. Our results imply that our intuition on what it means for a neural network to compress'' are not yet well understood.
  arXiv  Detail & Related papers  (2024-10-20T04:31:56Z)
• Sparse $L^1$-Autoencoders for Scientific Data Compression [0.0]
  We introduce effective data compression methods by developing autoencoders using high dimensional latent spaces that are $L1$-regularized. We show how these information-rich latent spaces can be used to mitigate blurring and other artifacts to obtain highly effective data compression methods for scientific data.
  arXiv  Detail & Related papers  (2024-05-23T07:48:00Z)
• Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth [83.15263499262824]
  We prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. We show how to improve upon Gaussian performance for the compression of sparse data by adding a denoising function to a shallow architecture. We validate our findings on image datasets, such as CIFAR-10 and MNIST.
  arXiv  Detail & Related papers  (2024-02-07T16:32:29Z)
• A Survey on Transformer Compression [84.18094368700379]
  Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV) Model compression methods reduce the memory and computational cost of Transformer. This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to Transformer-based models.
  arXiv  Detail & Related papers  (2024-02-05T12:16:28Z)
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods [91.54785981649228]
  This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
  arXiv  Detail & Related papers  (2022-12-27T12:37:34Z)
• Unrolled Compressed Blind-Deconvolution [77.88847247301682]
  sparse multichannel blind deconvolution (S-MBD) arises frequently in many engineering applications such as radar/sonar/ultrasound imaging. We propose a compression method that enables blind recovery from much fewer measurements with respect to the full received signal in time.
  arXiv  Detail & Related papers  (2022-09-28T15:16:58Z)
• An Introduction to Neural Data Compression [30.62305356319998]
  Neural compression is the application of neural networks and other machine learning methods to data compression. This introduction hopes to fill in the necessary background by reviewing basic coding topics such as entropy coding and rate-distortion theory.
  arXiv  Detail & Related papers  (2022-02-14T08:01:00Z)
• Neural Distributed Source Coding [59.630059301226474]
  We present a framework for lossy DSC that is agnostic to the correlation structure and can scale to high dimensions. We evaluate our method on multiple datasets and show that our method can handle complex correlations and state-of-the-art PSNR.
  arXiv  Detail & Related papers  (2021-06-05T04:50:43Z)
• Unfolding Neural Networks for Compressive Multichannel Blind Deconvolution [71.29848468762789]
  We propose a learned-structured unfolding neural network for the problem of compressive sparse multichannel blind-deconvolution. In this problem, each channel's measurements are given as convolution of a common source signal and sparse filter. We demonstrate that our method is superior to classical structured compressive sparse multichannel blind-deconvolution methods in terms of accuracy and speed of sparse filter recovery.
  arXiv  Detail & Related papers  (2020-10-22T02:34:33Z)
• Adaptive Estimators Show Information Compression in Deep Neural Networks [2.578242050187029]
  The information bottleneck theory proposes that neural networks achieve good generalization by compressing their representations to disregard information that is not relevant to the task. In this paper we develop more robust mutual information estimation techniques, that adapt to hidden activity of neural networks. We show that saturation of the activation function is not required for compression, and the amount of compression varies between different activation functions.
  arXiv  Detail & Related papers  (2019-02-24T23:41:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.