On the Relationship Between Representation Geometry and Generalization in Deep Neural Networks

• URL: http://arxiv.org/abs/2602.00130v1
• Date: Wed, 28 Jan 2026 04:33:41 GMT
• Title: On the Relationship Between Representation Geometry and Generalization in Deep Neural Networks
• Abstract summary: We show that effective dimension -- an unsupervised geometric metric -- strongly predicts accuracy.<n>We establish causality: degrading geometry via noise causes accuracy loss (r=-0.94, $p 10(9)$), while improving geometry via PCA maintains accuracy across architectures (-0.03pp at 95% variance)<n>These results establish that effective dimension provides domain-agnostic predictive and causal information about neural network performance, computed entirely without labels.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate the relationship between representation geometry and neural network performance. Analyzing 52 pretrained ImageNet models across 13 architecture families, we show that effective dimension -- an unsupervised geometric metric -- strongly predicts accuracy. Output effective dimension achieves partial r=0.75 ($p < 10^(-10)$) after controlling for model capacity, while total compression achieves partial r=-0.72. These findings replicate across ImageNet and CIFAR-10, and generalize to NLP: effective dimension predicts performance for 8 encoder models on SST-2/MNLI and 15 decoder-only LLMs on AG News (r=0.69, p=0.004), while model size does not (r=0.07). We establish bidirectional causality: degrading geometry via noise causes accuracy loss (r=-0.94, $p < 10^(-9)$), while improving geometry via PCA maintains accuracy across architectures (-0.03pp at 95% variance). This relationship is noise-type agnostic -- Gaussian, Uniform, Dropout, and Salt-and-pepper noise all show $|r| > 0.90$. These results establish that effective dimension provides domain-agnostic predictive and causal information about neural network performance, computed entirely without labels.

Related papers

• Optimal Pruning for Neural Architectures using Fisher Information Distances [0.15554979526414522]
  Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes.<n>The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space.<n>By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined.
  arXiv  Detail & Related papers  (2026-09-14T18:00:03Z)
• Efficient Hybrid CNN-GNN Architecture for Monocular Depth Estimation [0.0]
  We present GraphDepth, a monocular depth estimation architecture that integrates Graph Neural Networks (GNNs) within a convolutional encoder-decoder framework.<n>Our approach embeds efficient GraphSAGE layers at multiple scales of a ResNet-101 U-Net backbone.<n> Experiments on NYU Depth V2, WHU Aerial, ETH3D, and Mid-Air benchmarks demonstrate competitive accuracy within 4.6% of state-of-the-art transformers.
  arXiv  Detail & Related papers  (2026-05-11T09:21:04Z)
• Normalized Architectures are Natively 4-Bit [49.13186675123547]
  nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic.<n>We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters.
  arXiv  Detail & Related papers  (2026-05-07T11:54:07Z)
• Parameter-Efficient Architectural Modifications for Translation-Invariant CNNs [2.111646739533638]
  Convolutional Neural Networks (CNNs) are widely assumed to be translation-invariant, yet standard architectures exhibit a startling fragility.<n>We propose a lightweight 'Online Architecture' strategy by strategically inserting Global Average Pooling layers at various network depths.<n>This strategy achieves a massive 98% reduction in trainable parameters (from 5.2M to just 82K) and a 90% reduction in total network size.
  arXiv  Detail & Related papers  (2026-04-30T13:52:42Z)
• SAND: Spatially Adaptive Network Depth for Fast Sampling of Neural Implicit Surfaces [91.71563221166417]
  Implicit neural representations are powerful for geometric modeling, but their practical use is often limited by the high computational cost.<n>We propose an efficient neural implicit geometry representation framework with spatially adaptive network depth.
  arXiv  Detail & Related papers  (2026-04-15T14:14:17Z)
• Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement [0.0]
  Vision-language models encode geometry that their text pathway fails to express.<n>LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees.<n>These findings enable a single frozen backbone to function as a multi-task geometric sensor.
  arXiv  Detail & Related papers  (2026-03-06T16:48:27Z)
• Geometric Operator Learning with Optimal Transport [77.16909146519227]
  We propose integrating optimal transport (OT) into operator learning for partial differential equations (PDEs) on complex geometries.<n>For 3D simulations focused on surfaces, our OT-based neural operator embeds the surface geometry into a 2D parameterized latent space.<n> Experiments with Reynolds-averaged Navier-Stokes equations (RANS) on the ShapeNet-Car and DrivAerNet-Car datasets show that our method achieves better accuracy and also reduces computational expenses.
  arXiv  Detail & Related papers  (2025-07-26T21:28:25Z)
• MPruner: Optimizing Neural Network Size with CKA-Based Mutual Information Pruning [7.262751938473306]
  Pruning is a well-established technique that reduces the size of neural networks while mathematically guaranteeing accuracy preservation. We develop a new pruning algorithm, MPruner, that leverages mutual information through vector similarity. MPruner achieved up to a 50% reduction in parameters and memory usage for CNN and transformer-based models, with minimal to no loss in accuracy.
  arXiv  Detail & Related papers  (2024-08-24T05:54:47Z)
• Towards Generalized Entropic Sparsification for Convolutional Neural Networks [0.0]
  Convolutional neural networks (CNNs) are reported to be overparametrized. Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally-scalable entropic relaxation of the pruning problem. The sparse subnetwork is found from the pre-trained (full) CNN using the network entropy minimization as a sparsity constraint.
  arXiv  Detail & Related papers  (2024-04-06T21:33:39Z)
• Geometry-Informed Neural Operator for Large-Scale 3D PDEs [76.06115572844882]
  We propose the geometry-informed neural operator (GINO) to learn the solution operator of large-scale partial differential equations. We successfully trained GINO to predict the pressure on car surfaces using only five hundred data points.
  arXiv  Detail & Related papers  (2023-09-01T16:59:21Z)
• Combining Gradients and Probabilities for Heterogeneous Approximation of Neural Networks [2.5744053804694893]
  We discuss the validity of additive Gaussian noise as a surrogate model for behavioral simulation of approximate multipliers. The amount of noise injected into the accurate computations is learned during network training using backpropagation. Our experiments show that the combination of heterogeneous approximation and neural network retraining reduces the energy consumption for variants.
  arXiv  Detail & Related papers  (2022-08-15T15:17:34Z)
• The Mean Dimension of Neural Networks -- What causes the interaction effects? [0.9208007322096533]
  Owen and Hoyt recently showed that the effective dimension offers key structural information about the input-output mapping underlying an artificial neural network. This work proposes an estimation procedure that allows the calculation of the mean dimension from a given dataset.
  arXiv  Detail & Related papers  (2022-07-11T14:00:06Z)
• SmoothNets: Optimizing CNN architecture design for differentially private deep learning [69.10072367807095]
  DPSGD requires clipping and noising of per-sample gradients. This introduces a reduction in model utility compared to non-private training. We distilled a new model architecture termed SmoothNet, which is characterised by increased robustness to the challenges of DP-SGD training.
  arXiv  Detail & Related papers  (2022-05-09T07:51:54Z)
• Scaling Structured Inference with Randomization [64.18063627155128]
  We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
  arXiv  Detail & Related papers  (2021-12-07T11:26:41Z)
• Densely connected neural networks for nonlinear regression [8.830042935753303]
  We propose a novel DenseNet regression model where convolution and pooling layers are replaced by fully connected layers. The results give an optimal depth (19) and recommend a limited input dimension (under 200). Ultimately, DenseNet regression is applied to predict relative humidity, and the outcome shows a high correlation (0.91) with observations.
  arXiv  Detail & Related papers  (2021-07-29T03:41:56Z)
• Effective Model Sparsification by Scheduled Grow-and-Prune Methods [73.03533268740605]
  We propose a novel scheduled grow-and-prune (GaP) methodology without pre-training the dense models. Experiments have shown that such models can match or beat the quality of highly optimized dense models at 80% sparsity on a variety of tasks.
  arXiv  Detail & Related papers  (2021-06-18T01:03:13Z)
• Calibrating Deep Neural Networks using Focal Loss [77.92765139898906]
  Miscalibration is a mismatch between a model's confidence and its correctness. We show that focal loss allows us to learn models that are already very well calibrated. We show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases.
  arXiv  Detail & Related papers  (2020-02-21T17:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.