Critical Percolation as a Synthetic Data Model for Interpretability

• URL: http://arxiv.org/abs/2606.20347v1
• Date: Thu, 18 Jun 2026 15:15:57 GMT
• Title: Critical Percolation as a Synthetic Data Model for Interpretability
• Abstract summary: We introduce a family of synthetic datasets consisting of hierarchical functions defined on critical mean-field percolation clusters.<n>Latent variables modeling a taxonomic hierarchy generate each data point's target value.<n>We find that the model's ground-truth latent variables can be linearly decoded from neural network activations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks learn features that reflect the hierarchical, multi-scale structure of natural data. Synthetic datasets used to evaluate interpretability methods typically lack this structure, limiting their value as realistic toy models. To close this gap, we introduce a family of synthetic datasets consisting of hierarchical functions defined on critical mean-field percolation clusters embedded in a high-dimensional data space. The percolation data consists of sparse, low-dimensional fractal clusters with a power-law size distribution. Latent variables modeling a taxonomic hierarchy generate each data point's target value. The data model is analytically tractable with known critical exponents that fix its properties without requiring hyperparameter tuning. We leverage a mapping between percolation clusters, random trees, and additive coalescence to propose an almost linear-time algorithm to jointly sample a random tree and its hierarchical latent decomposition, enabling data generation at arbitrary scale. Using probing experiments, we find that the model's ground-truth latent variables can be linearly decoded from neural network activations. Together, sparsity, self-similarity, power-law statistics, and analytical tractability make critical percolation a principled testbed for interpretability research.

Related papers

• Active Learning with Low-Rank Structure for Data Selection [85.43185363043362]
  We introduce a new data selection framework based on low-rank approximation and residual-based sampling.<n>We show that it is possible to select a weighted subset of $tildeOleft(k + frac1varepsilon2right)$ data points whose average loss approximates the average loss over the full dataset.
  arXiv  Detail & Related papers  (2026-06-14T22:29:59Z)
• Synthics: Synthetic Physics-like Datasets for Machine Learning [0.18275108630751835]
  We generate synthetic regression datasets that structurally resemble physics equations from a given equation corpus.<n>The generated equations match the corpus on all of the eight studied structural features.<n>A gradient-boosted regressor tuned on the synthetic data picks, on average, the 6th-best configuration out of 20 on real data.
  arXiv  Detail & Related papers  (2026-06-04T21:20:08Z)
• Learning with Shallow Neural Networks on Cluster-Structured Features [13.080485957000464]
  We study how spatial correlations affect the sample complexity of learning with gradient descent on shallow neural networks.<n>We show that for a layerwise gradient-descent variant, the sample complexity scales with the number of hidden variables.<n>We empirically test our theoretical findings on both synthetic and real data.
  arXiv  Detail & Related papers  (2026-05-14T15:02:24Z)
• Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data [7.047721358838963]
  We study the theoretical behavior of denoising score matching.<n>We derive exact expressions for the test, train, and score errors in the high-dimensional limit.
  arXiv  Detail & Related papers  (2026-03-24T08:58:12Z)
• Diffusion Model for Manifold Data: Score Decomposition, Curvature, and Statistical Complexity [93.52679474816728]
  Diffusion models have become a leading framework in generative modeling.<n>This paper investigates how diffusion models learn structured data, focusing on two key aspects: statistical complexity and influence of data geometric properties.<n>We also highlight the interplay of manifold curvature with the structures in the score function.
  arXiv  Detail & Related papers  (2026-03-21T04:40:19Z)
• High-dimensional Analysis of Synthetic Data Selection [44.67519806837088]
  We show that, for linear models, the covariance shift between the target distribution and the distribution of the synthetic data affects the generalization error.<n>Remarkably, the theoretical insights from linear models carry over to deep neural networks and generative models.
  arXiv  Detail & Related papers  (2025-10-09T12:06:31Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
  Federated learning aims to train models across different clients without the sharing of data for privacy considerations. We study how data heterogeneity affects the representations of the globally aggregated models. We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
  arXiv  Detail & Related papers  (2022-10-01T09:04:17Z)
• Amortized Inference for Causal Structure Learning [72.84105256353801]
  Learning causal structure poses a search problem that typically involves evaluating structures using a score or independence test. We train a variational inference model to predict the causal structure from observational/interventional data. Our models exhibit robust generalization capabilities under substantial distribution shift.
  arXiv  Detail & Related papers  (2022-05-25T17:37:08Z)
• Learning Self-Expression Metrics for Scalable and Inductive Subspace Clustering [5.587290026368626]
  Subspace clustering has established itself as a state-of-the-art approach to clustering high-dimensional data. We propose a novel metric learning approach to learn instead a subspace affinity function using a siamese neural network architecture. Our model benefits from a constant number of parameters and a constant-size memory footprint, allowing it to scale to considerably larger datasets.
  arXiv  Detail & Related papers  (2020-09-27T15:40:12Z)
• Out-of-distribution Generalization via Partial Feature Decorrelation [72.96261704851683]
  We present a novel Partial Feature Decorrelation Learning (PFDL) algorithm, which jointly optimize a feature decomposition network and the target image classification model. The experiments on real-world datasets demonstrate that our method can improve the backbone model's accuracy on OOD image classification datasets.
  arXiv  Detail & Related papers  (2020-07-30T05:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.