Complexity of Linear Regions in Self-supervised Deep ReLU Networks

• URL: http://arxiv.org/abs/2604.24393v1
• Date: Mon, 27 Apr 2026 12:21:25 GMT
• Title: Complexity of Linear Regions in Self-supervised Deep ReLU Networks
• Abstract summary: This study investigates the local distribution of linear regions produced by Self-Supervised Learning (SSL) models.<n>We use SplineCam to extract two-dimensional polytopes near the data distribution.<n>The study compares supervised, contrastive, and self-distillation methods over two standard benchmark datasets.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: There has been growing interest in studying the complexity of Rectified Linear Unit (ReLU) based activation networks. Recent work investigates the evolution of the number of piecewise-linear partitions (linear regions) that are formed during training. However, current research is limited to examining the complexity of models trained in a supervised way. Self-Supervised Learning (SSL) differs in that it directly optimises the representation space using a loss function to enhance the model's performance across multiple downstream tasks. This study investigates the local distribution of linear regions produced by SSL models. We demonstrate that the evolution of linear regions correlates with the representation quality by utilising SplineCam to extract two-dimensional polytopes near the data distribution. We track the number, area, eccentricity, and boundaries of regions throughout training. The study compares supervised, contrastive, and self-distillation methods over two standard benchmark datasets, MNIST and FashionMNIST. The analysis of the experimental results shows that self-supervised methods create substantially fewer regions to achieve comparable accuracy to supervised models. Contrastive methods rapidly expand regions over time, whereas self-distillation methods tend to consolidate by merging neighbouring regions. Lastly, we can detect representation collapse early within the geometric space of linear regions. Our analysis suggests that polytopal metrics can serve as reliable indicators of representation quality and model performance.

Related papers

• Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity [5.366707531676941]
  Piecewise-affine neural networks implement continuous piecewise-affine maps.<n>We study the gap between theoretical region capacity and the affine regions realized after training.
  arXiv  Detail & Related papers  (2026-06-19T18:55:46Z)
• Region Seeding via Pre-Activation Regularization: A Geometric View of Piecewise Affine Neural Networks [12.476368894025233]
  In practice, standard training realizes far fewer region refinements in data-visited neighborhoods than the architecture could in principle support.<n>Our theory provides a sufficient condition under which bringing neuron switching surfaces sufficiently close to data points ensures their intersection with local neighborhoods.<n>We propose a plug-and-play region-seeding regularizer that encourages early partitioning while allowing task-driven refinement to dominate later in training.
  arXiv  Detail & Related papers  (2026-05-07T14:05:01Z)
• Divide-and-Conquer Approach to Holistic Cognition in High-Similarity Contexts with Limited Data [73.20753799870282]
  We propose a novel Divide-and-Conquer Holistic Cognition Network (DHCNet)<n>DHCNet implements a divide-and-conquer strategy by decomposing holistic cues into spatially-associated subtle discrepancies.<n>DHCNet achieves remarkable performance on five widely-used Ultra-FGVC datasets.
  arXiv  Detail & Related papers  (2026-04-21T11:17:09Z)
• Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistency [52.52950138164424]
  We show that when leveraging the off-the-shelf (vision) foundation models for feature extraction, the geometric shapes of the resulting feature distributions exhibit remarkable transferability across domains and datasets.<n>We embody our geometric knowledge-guided distribution calibration framework in two popular and challenging settings: federated learning and long-tailed recognition.<n>In long-tailed learning, it utilizes the geometric knowledge transferred from sample-rich categories to recover the true distribution for sample-scarce tail classes.
  arXiv  Detail & Related papers  (2025-08-19T05:22:59Z)
• Distributed Gradient Descent with Many Local Steps in Overparameterized Models [20.560882414631784]
  In distributed training of machine learning models, gradient descent with local iterative steps is a popular method.<n>We try to explain this good performance from a viewpoint of implicit bias in Local Gradient Descent (Local-GD) with a large number of local steps.
  arXiv  Detail & Related papers  (2024-12-10T23:19:40Z)
• Certified Training with Branch-and-Bound: A Case Study on Lyapunov-stable Neural Control [64.58719561861079]
  We develop a new and generally formulated certified training framework named CT-BaB.<n>In order to handle the relatively large region-of-interest, we propose a novel framework of training-time branch-and-bound.<n>We demonstrate that our new training framework can produce models which can be more efficiently verified at test time.
  arXiv  Detail & Related papers  (2024-11-27T11:12:46Z)
• Learning Regionalization using Accurate Spatial Cost Gradients within a Differentiable High-Resolution Hydrological Model: Application to the French Mediterranean Region [0.18139022013189662]
  Estimating distributed hydrological parameters in ungauged catchments poses a challenging regionalization problem. This paper introduces a Hybrid Assimilation and Regionalization (HDA-PR) approach incorporating learnable regionalization mappings. Results highlight a strong regionalization of HDA-PR especially in the most challenging upstream-to-downstream extrapolation scenario.
  arXiv  Detail & Related papers  (2023-08-02T07:23:50Z)
• Reconstructing Spatiotemporal Data with C-VAEs [49.1574468325115]
  Conditional continuous representation of moving regions is commonly used. In this work, we explore the capabilities of Conditional Varitemporal Autoencoder (C-VAE) models to generate realistic representations of regions' evolution.
  arXiv  Detail & Related papers  (2023-07-12T15:34:10Z)
• CHALLENGER: Training with Attribution Maps [63.736435657236505]
  We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance. In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
  arXiv  Detail & Related papers  (2022-05-30T13:34:46Z)
• DISCO Verification: Division of Input Space into COnvex polytopes for neural network verification [0.0]
  The impressive results of modern neural networks partly come from their non linear behaviour. We propose a method to simplify the verification problem by operating a partitionning into multiple linear subproblems. We also present the impact of a technique aiming at reducing the number of linear regions during training.
  arXiv  Detail & Related papers  (2021-05-17T12:40:51Z)
• Domain-Adversarial Training of Self-Attention Based Networks for Land Cover Classification using Multi-temporal Sentinel-2 Satellite Imagery [0.0]
  Most practical applications cannot rely on labeled data, and in the field, surveys are a time consuming solution. In this paper, we investigate adversarial training of deep neural networks to bridge the domain discrepancy between distinct geographical zones.
  arXiv  Detail & Related papers  (2021-04-01T15:45:17Z)
• GELATO: Geometrically Enriched Latent Model for Offline Reinforcement Learning [54.291331971813364]
  offline reinforcement learning approaches can be divided into proximal and uncertainty-aware methods. In this work, we demonstrate the benefit of combining the two in a latent variational model. Our proposed metrics measure both the quality of out of distribution samples as well as the discrepancy of examples in the data.
  arXiv  Detail & Related papers  (2021-02-22T19:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.