Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

• URL: http://arxiv.org/abs/2604.10560v1
• Date: Sun, 12 Apr 2026 10:04:48 GMT
• Title: Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria
• Abstract summary: Profiled Sparse Networks (PSN) replace uniform connectivity with deterministic, heterogeneous fan-in profiles.<n>At 90% sparsity, all static profiles achieve accuracy within 0.2-0.6% of dense baselines on every dataset.<n>This result holds across sparsity levels (80-99.9%), profile shapes (eight parametric families, lognormal, and power-law), and fan-in coefficients of variation from 0 to 2.5.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Profiled Sparse Networks (PSN) replace uniform connectivity with deterministic, heterogeneous fan-in profiles defined by continuous, nonlinear functions, creating neurons with both dense and sparse receptive fields. We benchmark PSN across four classification datasets spanning vision and tabular domains, input dimensions from 54 to 784, and network depths of 2--3 hidden layers. At 90% sparsity, all static profiles, including the uniform random baseline, achieve accuracy within 0.2-0.6% of dense baselines on every dataset, demonstrating that heterogeneous connectivity provides no accuracy advantage when hub placement is arbitrary rather than task-aligned. This result holds across sparsity levels (80-99.9%), profile shapes (eight parametric families, lognormal, and power-law), and fan-in coefficients of variation from 0 to 2.5. Internal gradient analysis reveals that structured profiles create a 2-5x gradient concentration at hub neurons compared to the ~1x uniform distribution in random baselines, with the hierarchy strength predicted by fan-in coefficient of variation ($r = 0.93$). When PSN fan-in distributions are used to initialise RigL dynamic sparse training, lognormal profiles matched to the equilibrium fan-in distribution consistently outperform standard ERK initialisation, with advantages growing on harder tasks, achieving +0.16% on Fashion-MNIST ($p = 0.036$, $d = 1.07$), +0.43% on EMNIST, and +0.49% on Forest Cover. RigL converges to a characteristic fan-in distribution regardless of initialisation. Starting at this equilibrium allows the optimiser to refine weights rather than rearrange topology. Which neurons become hubs matters more than the degree of connectivity variance, i.e., random hub placement provides no advantage, while optimisation-driven placement does.

Related papers

• Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems [0.2730969268472861]
  Kolmogorov-Arnold Networks (KANs) replace the fixed activation functions and linear weights of Multi-Layer Perceptrons (MLPs)<n>This paper presents an empirical scalability study of data-parallel KAN training on multi-node, multi-GPU high-performance computing infrastructure.<n>Experiments were conducted on the FinisTerrae III supercomputer using up to 8 NVIDIA A100 GPUs across 4 nodes with PyTorch Distributed Data Parallel (DDP)
  arXiv  Detail & Related papers  (2026-09-07T16:45:20Z)
• SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits [6.860988566886594]
  We present textbfSparseKAN, a unified approach that compresses KANs along three complementary axes: basis functions, neurons/channels, and numerical precision.<n>SparseKAN equips the base branch, nonlinear basis branch, and individual basis terms with hierarchical learnable gates trained under a differentiable active-cost objective.
  arXiv  Detail & Related papers  (2026-08-01T20:42:41Z)
• SEED: Targeted Data Selection by Weighted Independent Set [76.68391670109433]
  We develop a robust and scalable data selection pipeline dubbed SEED.<n>SEED consistently outperforms state-of-the-art methods on instruction tuning, visual instruction tuning, and semantic segmentation.
  arXiv  Detail & Related papers  (2026-05-15T07:26:54Z)
• FI-KAN: Fractal Interpolation Kolmogorov-Arnold Networks [0.0]
  Kolmogorov-Arnold Networks (KAN) employ B-spline bases on a fixed grid, providing no intrinsic multi-scale decomposition for non-smooth function approximation.<n>We introduce Fractal Interpolation KAN (FI-KAN), which incorporates learnable fractal function (FIF) bases from iterated function system (IFS) theory into KAN.<n>Two variants are presented: Pure FI-KAN replaces B-splines entirely with FIF bases; Hybrid FI-KAN retains the B-spline path and adds a learnable fractal correction.
  arXiv  Detail & Related papers  (2026-03-30T11:09:07Z)
• The Coordinate System Problem in Persistent Structural Memory for Neural Architectures [0.0]
  We introduce the Dual-View Pheromone Pathway Network (DPPN), an architecture that routes sparse attention through a persistent pheromone field over latent slot transitions.<n>We identify a core principle: persistent memory requires a stable coordinate system.<n>We show that neither contrastive updates, multi-source distillation, Hungarian alignment, nor semantic decomposition resolves the instability when embeddings are learned from scratch.
  arXiv  Detail & Related papers  (2026-03-24T06:51:59Z)
• Spline Dimensional Decomposition with Interpolation-based Optimal Knot Selection for Stochastic Dynamic Analysis [6.407952035735354]
  Spline dimensional (SDD) addresses non-smooth or locally nonlinear behaviors by partitioning input coordinates via knot placement.<n>We propose a computationally efficient, computationally expensive method for optimal knot selection in SDD.<n>A modal analysis of a lower control arm shows that SDD with the proposed knots yields higher accuracy than SDD with uniformly or randomly spaced knots.
  arXiv  Detail & Related papers  (2025-05-19T09:08:39Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Bounding the Width of Neural Networks via Coupled Initialization -- A Worst Case Analysis [121.9821494461427]
  We show how to significantly reduce the number of neurons required for two-layer ReLU networks. We also prove new lower bounds that improve upon prior work, and that under certain assumptions, are best possible.
  arXiv  Detail & Related papers  (2022-06-26T06:51:31Z)
• Predicting the Stability of Hierarchical Triple Systems with Convolutional Neural Networks [68.8204255655161]
  We propose a convolutional neural network model to predict the stability of hierarchical triples. All trained models are made publicly available, allowing to predict the stability of hierarchical triple systems $200$ times faster than pure $N$-body methods.
  arXiv  Detail & Related papers  (2022-06-24T17:58:13Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Decentralized Sparse Linear Regression via Gradient-Tracking: Linear Convergence and Statistical Guarantees [23.256961881716595]
  We study a sparse linear regression over a network of agents, modeled as an undirected graph and no server node.<n>We analyze the convergence rate and statistical guarantees of a distributed projected gradient tracking-based algorithm.
  arXiv  Detail & Related papers  (2022-01-21T01:26:08Z)
• Federated Functional Gradient Boosting [75.06942944563572]
  We study functional minimization in Federated Learning. For both FFGB.C and FFGB.L, the radii of convergence shrink to zero as the feature distributions become more homogeneous.
  arXiv  Detail & Related papers  (2021-03-11T21:49:19Z)
• Optimizing Mode Connectivity via Neuron Alignment [84.26606622400423]
  Empirically, the local minima of loss functions can be connected by a learned curve in model space along which the loss remains nearly constant. We propose a more general framework to investigate effect of symmetry on landscape connectivity by accounting for the weight permutations of networks being connected.
  arXiv  Detail & Related papers  (2020-09-05T02:25:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.