Geometric and Information Compression of Representations in Deep Learning

• URL: http://arxiv.org/abs/2606.21593v2
• Date: Thu, 25 Jun 2026 13:10:27 GMT
• Title: Geometric and Information Compression of Representations in Deep Learning
• Abstract summary: Deep neural networks transform input data into latent representations that support a wide range of downstream tasks.<n>A central open question is whether low mutual information (MI) between inputs and representations necessarily implies geometrically compressed latent spaces.<n>We investigate this question using class-wise clustering as a measure of geometric compression and theoretically sound MI estimation.
• Score: 17.830589700881507
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks transform input data into latent representations that support a wide range of downstream tasks. These representations can be characterized along information-theoretic and geometric dimensions, but their relationship remains poorly understood. A central open question is whether low mutual information (MI) between inputs and representations necessarily implies geometrically compressed latent spaces and vice versa. We investigate this question using class-wise clustering as a measure of geometric compression and theoretically sound MI estimation in conditional entropy bottleneck (CEB) networks and continuous dropout networks. We evaluate the interplay between MI, geometric compression, and generalization on classification tasks under controlled noise injection schemes. Our findings show that low MI does not reliably correspond to geometric compression, and that the connection between the two is more nuanced than often assumed. Indeed, our experiments reveal a negative and nonlinear relationship that can reverse when varying training setup. Our results put forward a hypothesis that generalization acts as a potential confounder in this connection rather than being their direct consequence.

Related papers

• Information Plane Analysis of Binary Neural Networks [6.259398755176288]
  We perform IP analyses on binary neural networks (BNNs) where activations are discrete and MI is finite.<n>We train BNNs to investigate the existence of late-stage compression phases and the relationship between compressed representations and generalisation performance.
  arXiv  Detail & Related papers  (2026-05-05T11:08:18Z)
• Generalization Below the Edge of Stability: The Role of Data Geometry [60.147710896851045]
  We show how data geometry controls generalization in ReLU networks trained below the edge of stability.<n>For data distributions supported on a mixture of low-dimensional balls, we derive generalization bounds that provably adapt to the intrinsic dimension.<n>Our results consolidate disparate empirical findings that have appeared in the literature.
  arXiv  Detail & Related papers  (2025-10-20T21:40:36Z)
• Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion [56.12939353271623]
  From a geometric standpoint, the word representation space of highly compressed LMs tends to degenerate into a highly anisotropic state.<n>We find this synchronicity is essentially the Compression Hacking'' in LM representations.<n>We propose three refined compression metrics by incorporating geometric distortion analysis and integrate them into a self-evaluation pipeline.
  arXiv  Detail & Related papers  (2025-05-23T12:11:03Z)
• Learning to Compress: Local Rank and Information Compression in Deep Neural Networks [2.483371671716944]
  Deep neural networks tend to exhibit a bias toward low-rank solutions during training, implicitly learning low-dimensional feature representations. We introduce the concept of local rank as a measure of feature manifold dimensionality and demonstrate that this rank decreases during the final phase of training. We argue that networks that reduce the rank of their learned representations also compress mutual information between inputs and intermediate layers.
  arXiv  Detail & Related papers  (2024-10-10T07:53:34Z)
• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• A Geometrical Approach to Evaluate the Adversarial Robustness of Deep Neural Networks [52.09243852066406]
  Adversarial Converging Time Score (ACTS) measures the converging time as an adversarial robustness metric. We validate the effectiveness and generalization of the proposed ACTS metric against different adversarial attacks on the large-scale ImageNet dataset.
  arXiv  Detail & Related papers  (2023-10-10T09:39:38Z)
• A simple connection from loss flatness to compressed neural representations [3.5502600490147196]
  Sharpness, a geometric measure in the parameter space that reflects the flatness of the loss landscape, has long been studied for its potential connections to neural network behavior.<n>In this paper, we investigate how sharpness influences the local geometric features of neural representations in feature space.
  arXiv  Detail & Related papers  (2023-10-03T03:36:29Z)
• Understanding the Effect of the Long Tail on Neural Network Compression [9.819486253052528]
  We study the "long tail" phenomenon in computer vision datasets observed by Feldman, et al. As compression limits the capacity of a network (and hence also its ability to memorize), we study the question: are mismatches between the full and compressed models correlated with the memorized training data?
  arXiv  Detail & Related papers  (2023-06-09T20:18:05Z)
• Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression [37.69303106863453]
  The Information Bottleneck (IB) principle offers an information-theoretic framework for analyzing the training process of deep neural networks (DNNs) In this paper, we introduce a framework for conducting IB analysis of general NNs. We also perform IB analysis on a close-to-real-scale, which reveals new features of the MI dynamics.
  arXiv  Detail & Related papers  (2023-05-13T21:44:32Z)
• Information Bottleneck: Exact Analysis of (Quantized) Neural Networks [16.199954215341833]
  The information bottleneck (IB) principle has been suggested as a way to analyze deep neural networks. Separate fitting and compression phases during training have been reported. Our study shows that the initial IB results were not artifacts of binning when computing the MI.
  arXiv  Detail & Related papers  (2021-06-24T11:13:08Z)
• MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning [90.20563679417567]
  This paper introduces MDP homomorphic networks for deep reinforcement learning. MDP homomorphic networks are neural networks that are equivariant under symmetries in the joint state-action space of an MDP. We show that such networks converge faster than unstructured networks on CartPole, a grid world and Pong.
  arXiv  Detail & Related papers  (2020-06-30T15:38:37Z)
• Understanding Generalization in Deep Learning via Tensor Methods [53.808840694241]
  We advance the understanding of the relations between the network's architecture and its generalizability from the compression perspective. We propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks.
  arXiv  Detail & Related papers  (2020-01-14T22:26:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.