Storage capacity of perceptron with variable selection

• URL: http://arxiv.org/abs/2512.01861v1
• Date: Mon, 01 Dec 2025 16:44:57 GMT
• Title: Storage capacity of perceptron with variable selection
• Authors: Yingying Xu, Masayuki Ohzeki, Yoshiyuki Kabashima,
• Abstract summary: A central challenge in machine learning is to distinguish genuine structure from chance correlations in high-dimensional data.<n>We show that a simple perceptron can perfectly classify $P = N$ random patterns by optimally selecting $M = N$ variables out of $N$ variables.<n>This not only provides a quantitative criterion for distinguishing true structure in the data from spurious regularities, but also yields the storage capacity of associative memory models.
• Score: 10.64866985260943
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A central challenge in machine learning is to distinguish genuine structure from chance correlations in high-dimensional data. In this work, we address this issue for the perceptron, a foundational model of neural computation. Specifically, we investigate the relationship between the pattern load $α$ and the variable selection ratio $ρ$ for which a simple perceptron can perfectly classify $P = αN$ random patterns by optimally selecting $M = ρN$ variables out of $N$ variables. While the Cover--Gardner theory establishes that a random subset of $ρN$ dimensions can separate $αN$ random patterns if and only if $α< 2ρ$, we demonstrate that optimal variable selection can surpass this bound by developing a method, based on the replica method from statistical mechanics, for enumerating the combinations of variables that enable perfect pattern classification. This not only provides a quantitative criterion for distinguishing true structure in the data from spurious regularities, but also yields the storage capacity of associative memory models with sparse asymmetric couplings.

Related papers

• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Learning Compositional Functions with Transformers from Easy-to-Hard Data [63.96562216704653]
  We study the learnability of the $k$-fold composition task, which requires computing an interleaved composition of $k$ input permutations and $k$ hidden permutations.<n>We show that this function class can be efficiently learned, with runtime and sample in $k$, by gradient descent on an $O(log k)$-depth transformer.
  arXiv  Detail & Related papers  (2025-05-29T17:22:00Z)
• Is Supervised Learning Really That Different from Unsupervised? [12.424687747519037]
  We demonstrate how supervised learning can be decomposed into a two-stage procedure.<n>We show that versions of linear and kernel ridge regression, smoothing splines, and neural networks, which are trained without access to y, perform similarly to their standard y-based counterparts.
  arXiv  Detail & Related papers  (2025-05-16T08:51:44Z)
• Let's Think Var-by-Var: Large Language Models Enable Ad Hoc Probabilistic Reasoning [15.568698101627088]
  We propose to extract common sense from large language models (LLMs)<n>We focus our investigation on $textitguesstimation$ questions such as "How much are Airbnb listings in Newark, NJ?"<n>Our framework answers such a question by an $textitad hoc$ probabilistic model.
  arXiv  Detail & Related papers  (2024-12-03T01:53:06Z)
• Computational-Statistical Gaps in Gaussian Single-Index Models [77.1473134227844]
  Single-Index Models are high-dimensional regression problems with planted structure. We show that computationally efficient algorithms, both within the Statistical Query (SQ) and the Low-Degree Polynomial (LDP) framework, necessarily require $Omega(dkstar/2)$ samples.
  arXiv  Detail & Related papers  (2024-03-08T18:50:19Z)
• EM for Mixture of Linear Regression with Clustered Data [6.948976192408852]
  We discuss how the underlying clustered structures in distributed data can be exploited to improve learning schemes. We employ the well-known Expectation-Maximization (EM) method to estimate the maximum likelihood parameters from $m$ batches of dependent samples. We show that if properly, EM on the structured data requires only $O(1)$ to reach the same statistical accuracy, as long as $m$ grows up as $eo(n)$.
  arXiv  Detail & Related papers  (2023-08-22T15:47:58Z)
• Conformalization of Sparse Generalized Linear Models [2.1485350418225244]
  Conformal prediction method estimates a confidence set for $y_n+1$ that is valid for any finite sample size. Although attractive, computing such a set is computationally infeasible in most regression problems. We show how our path-following algorithm accurately approximates conformal prediction sets.
  arXiv  Detail & Related papers  (2023-07-11T08:36:12Z)
• On the Identifiability and Estimation of Causal Location-Scale Noise Models [122.65417012597754]
  We study the class of location-scale or heteroscedastic noise models (LSNMs) We show the causal direction is identifiable up to some pathological cases. We propose two estimators for LSNMs: an estimator based on (non-linear) feature maps, and one based on neural networks.
  arXiv  Detail & Related papers  (2022-10-13T17:18:59Z)
• $p$-Generalized Probit Regression and Scalable Maximum Likelihood Estimation via Sketching and Coresets [74.37849422071206]
  We study the $p$-generalized probit regression model, which is a generalized linear model for binary responses. We show how the maximum likelihood estimator for $p$-generalized probit regression can be approximated efficiently up to a factor of $(1+varepsilon)$ on large data.
  arXiv  Detail & Related papers  (2022-03-25T10:54:41Z)
• Efficient and robust high-dimensional sparse logistic regression via nonlinear primal-dual hybrid gradient algorithms [0.0]
  We propose an iterative algorithm that provably computes a solution to a logistic regression problem regularized by an elastic net penalty. This result improves on the known complexity bound of $O(min(m2n,mn2)log (1/epsilon))$ for first-order optimization methods.
  arXiv  Detail & Related papers  (2021-11-30T14:16:48Z)
• Improving Robustness and Generality of NLP Models Using Disentangled Representations [62.08794500431367]
  Supervised neural networks first map an input $x$ to a single representation $z$, and then map $z$ to the output label $y$. We present methods to improve robustness and generality of NLP models from the standpoint of disentangled representation learning. We show that models trained with the proposed criteria provide better robustness and domain adaptation ability in a wide range of supervised learning tasks.
  arXiv  Detail & Related papers  (2020-09-21T02:48:46Z)
• Neural Bayes: A Generic Parameterization Method for Unsupervised Representation Learning [175.34232468746245]
  We introduce a parameterization method called Neural Bayes. It allows computing statistical quantities that are in general difficult to compute. We show two independent use cases for this parameterization.
  arXiv  Detail & Related papers  (2020-02-20T22:28:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.