Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity

• URL: http://arxiv.org/abs/2606.21687v1
• Date: Fri, 19 Jun 2026 18:55:46 GMT
• Title: Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity
• Abstract summary: Piecewise-affine neural networks implement continuous piecewise-affine maps.<n>We study the gap between theoretical region capacity and the affine regions realized after training.
• Score: 5.366707531676941
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Piecewise-affine neural networks (e.g., with ReLU or LeakyReLU activations) implement continuous piecewise-affine maps, and the number of affine regions provides a natural proxy for expressive capacity. However, the gap between theoretical region capacity and the affine regions realized after training remains insufficiently understood. We study this gap from two complementary perspectives. First, we give a rigorous, architecture-dependent theorem for affine line-segment probes: for multilayer perceptrons with piecewise-affine activations, the number of affine pieces realized along an affine line-segment probe is upper bounded by an explicit product of layer-wise width terms (and activation breakpoint factors). This yields a neuron-threshold lower bound for representing target functions with prescribed one-dimensional piece complexity, formalizing the minimal region budget required for complex signals. Second, we exactly enumerate affine regions realized within bounded 2D and higher-dimensional domains under controlled task complexity. Under fixed architectures and training protocols, increasing input--label complexity yields trained solutions with markedly fewer realized regions in the evaluation domain, even though worst-case architectural capacity is unchanged; we call this reduced region usage expressivity saturation. Moreover, in the most challenging regimes, 2D visualizations show that region-usage collapse often coincides with degraded decision boundaries. Finally, we visualize the training dynamics of affine-region partitions and decision boundaries, revealing a consistent refinement process during optimization.

Related papers

• Characterizing the Discrete Geometry of ReLU Networks [4.716880809392674]
  ReLU networks define continuous piecewise-linear functions and their linear regions are polyhedra in the input space.<n>We show that the way these regions fit together is fundamental to the behavior of the network.<n>We prove new theoretical results about these complexes that hold for all fully-connected ReLU networks.
  arXiv  Detail & Related papers  (2026-06-05T17:36:19Z)
• AffineLens: Capturing the Continuous Piecewise Affine Functions of Neural Networks [23.025211488990077]
  Piecewise affine neural networks (PANNs) provide a principled geometric perspective on neural network expressivity.<n>We present AffineLens, a unified framework for computing the hyperplane arrangements and polyhedral structures underlying PANNs.
  arXiv  Detail & Related papers  (2026-05-07T13:16:39Z)
• SAND: Spatially Adaptive Network Depth for Fast Sampling of Neural Implicit Surfaces [91.71563221166417]
  Implicit neural representations are powerful for geometric modeling, but their practical use is often limited by the high computational cost.<n>We propose an efficient neural implicit geometry representation framework with spatially adaptive network depth.
  arXiv  Detail & Related papers  (2026-04-15T14:14:17Z)
• A Dual Certificate Approach to Sparsity in Infinite-Width Shallow Neural Networks [1.516610503825416]
  We study total variation (TV)-regularized training of infinite-width shallow ReLU neural networks, formulated as a convex optimization problem over measures on the unit sphere.<n>Our approach leverages the duality theory of TV-regularized optimization problems to establish rigorous guarantees on the sparsity of the solutions to the training problem.
  arXiv  Detail & Related papers  (2026-03-18T14:46:55Z)
• Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning [50.88504784466931]
  Multi-task dense prediction involves semantic segmentation, depth estimation, and surface normal estimation. Existing solutions typically rely on learning global image representations for global cross-task image matching. Our proposal involves modeling region-wise representations using Gaussian Distributions.
  arXiv  Detail & Related papers  (2024-03-15T12:41:30Z)
• On the Number of Regions of Piecewise Linear Neural Networks [16.78532039510369]
  Many feedforward neural networks (NNs) generate continuous and piecewise-linear (CPWL) mappings. The number of these so-called linear regions offers a natural metric to characterize the expressiveness of CPWL NNs. We introduce a complementary framework to estimate the average number of linear regions produced by a CPWL NN.
  arXiv  Detail & Related papers  (2022-06-17T08:17:28Z)
• Region-Based Semantic Factorization in GANs [67.90498535507106]
  We present a highly efficient algorithm to factorize the latent semantics learned by Generative Adversarial Networks (GANs) concerning an arbitrary image region. Through an appropriately defined generalized Rayleigh quotient, we solve such a problem without any annotations or training. Experimental results on various state-of-the-art GAN models demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2022-02-19T17:46:02Z)
• Sign-Agnostic CONet: Learning Implicit Surface Reconstructions by Sign-Agnostic Optimization of Convolutional Occupancy Networks [39.65056638604885]
  We learn implicit surface reconstruction by sign-agnostic optimization of convolutional occupancy networks. We show this goal can be effectively achieved by a simple yet effective design.
  arXiv  Detail & Related papers  (2021-05-08T03:35:32Z)
• InverseForm: A Loss Function for Structured Boundary-Aware Segmentation [80.39674800972182]
  We present a novel boundary-aware loss term for semantic segmentation using an inverse-transformation network. This plug-in loss term complements the cross-entropy loss in capturing boundary transformations. We analyze the quantitative and qualitative effects of our loss function on three indoor and outdoor segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-04-06T18:52:45Z)
• Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks [3.04585143845864]
  Several current bounds on the maximal number of affine regions of a ReLU feed-forward neural network are special cases. We analyze and partially solve a problem in algebraic topology the solution of which would fully exploit this framework.
  arXiv  Detail & Related papers  (2021-03-31T15:50:44Z)
• Hierarchical Verification for Adversarial Robustness [89.30150585592648]
  We introduce a new framework for the exact point-wise $ell_p$ robustness verification problem. LayerCert exploits the layer-wise geometric structure of deep feed-forward networks with rectified linear activations (ReLU networks) We show that LayerCert provably reduces the number and size of the convex programs that one needs to solve compared to GeoCert.
  arXiv  Detail & Related papers  (2020-07-23T07:03:05Z)
• Local Propagation in Constraint-based Neural Network [77.37829055999238]
  We study a constraint-based representation of neural network architectures. We investigate a simple optimization procedure that is well suited to fulfil the so-called architectural constraints.
  arXiv  Detail & Related papers  (2020-02-18T16:47:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.