Learning from Noise: Effective-Rank Collapse and Out-of-Distribution Rejection in Restricted Boltzmann Machines

• URL: http://arxiv.org/abs/2607.10506v1
• Date: Sat, 11 Jul 2026 23:31:52 GMT
• Title: Learning from Noise: Effective-Rank Collapse and Out-of-Distribution Rejection in Restricted Boltzmann Machines
• Abstract summary: We analyze the failure mode through the spectrum of the induced visible--visible interaction $J=WWT$.<n>We show that random auxiliary exposure can reshape both the interaction spectrum and the free-energy landscape of an energy-based classifier.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Restricted Boltzmann machines (RBMs) represent data by shaping an energy landscape over visible and hidden configurations, but their discriminative use is fragile under out-of-distribution (OOD) inputs: samples outside the training distribution can be absorbed into one of the learned class basins rather than rejected. Here, we analyze this failure mode through the spectrum of the induced visible--visible interaction $J=WW^{T}$, where \(W\) is the visible--hidden weight matrix. Relative to a Marchenko--Pastur random-matrix reference, conventional training spreads spectral weight into many weak, bulk-compatible directions, increasing the effective rank of $J$. When auxiliary random binary images are assigned to a rejection label during training, the learned interaction undergoes effective-rank collapse: weak bulk-like modes are depleted, spectral weight concentrates into fewer dominant eigendirections, and the effective rank of $J$ approaches that of the empirical data covariance matrix. The resulting RBM rejects structured OOD image datasets while preserving MNIST classification accuracy, showing that random auxiliary exposure can reshape both the interaction spectrum and the free-energy landscape of an energy-based classifier.

Related papers

• From Latent Space to Training Data: Explainable Specialization in Minimal MLPs [0.47554242259005197]
  We study whether training biases can make hidden neurons specialize in minimal one-hidden-layers.<n>Experiments on uniformly sampled one-dimensional datasets show a stable pattern from N = 3 to N = 100 across 480 controlled runs.
  arXiv  Detail & Related papers  (2026-05-25T15:18:22Z)
• Anchoring the Eigengap: Cross-Modal Spectral Stabilization for Sample-Efficient Representation Learning [0.8326214193432122]
  We show that vision models degrade sharply in low-data regimes, particularly in medical imaging where labeled samples are scarce.<n>We develop a theory of finite-sample representation learning that quantifies the recoverable dimension K(N)<n>We show that multimodal training maintains more stable modes and improves class separation, even when unimodal models achieve comparable few-shot accuracy.
  arXiv  Detail & Related papers  (2026-05-09T07:49:43Z)
• Hidden Unit Interpretability in RBM Quantum States:Encoding Antiferromagnetic Order in Heisenberg Spin Rings [0.0]
  We show that individual hidden units spontaneously specialize to capture staggered magnetization patterns characteristic of antiferromagnetic ground states.<n>No individual hidden unit reproduces the full antiferromagnetic correlations, indicating that quantum order emerges through collective encoding.
  arXiv  Detail & Related papers  (2026-03-25T11:51:59Z)
• SIGMA: Scalable Spectral Insights for LLM Collapse [51.863164847253366]
  We introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework for model collapse.<n>By utilizing benchmarks that deriving and deterministic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space.<n>We demonstrate that SIGMA effectively captures the transition towards states, offering both theoretical insights into the mechanics of collapse.
  arXiv  Detail & Related papers  (2026-01-06T19:47:11Z)
• Disordered Dynamics in High Dimensions: Connections to Random Matrices and Machine Learning [52.26396748560348]
  We provide an overview of high dimensional dynamical systems driven by random matrices.<n>We focus on applications to simple models of learning and generalization in machine learning theory.
  arXiv  Detail & Related papers  (2026-01-03T00:12:32Z)
• From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications [85.17672240603011]
  We study the non-uniform low-rank properties of weight matrices in Large Language Models.<n>We present Weight Low-Rank Projection (WeLore) that unifies weight compression and memory-efficient fine-tuning into one.
  arXiv  Detail & Related papers  (2024-07-15T21:05:20Z)
• Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification [74.62203971625173]
  Imbalanced data pose challenges for deep learning based classification models. One of the most widely-used approaches for tackling imbalanced data is re-weighting. We propose a novel re-weighting method based on optimal transport (OT) from a distributional point of view.
  arXiv  Detail & Related papers  (2022-08-05T01:23:54Z)
• An Embedding-Dynamic Approach to Self-supervised Learning [8.714677279673738]
  We treat the embeddings of images as point particles and consider model optimization as a dynamic process on this system of particles. Our dynamic model combines an attractive force for similar images, a locally dispersive force to avoid local collapse, and a global dispersive force to achieve a globally-homogeneous distribution of particles.
  arXiv  Detail & Related papers  (2022-07-07T19:56:20Z)
• Fast Ensemble Learning Using Adversarially-Generated Restricted Boltzmann Machines [0.0]
  Restricted Boltzmann Machine (RBM) has received recent attention and relies on an energy-based structure to model data probability distributions. This work proposes to artificially generate RBMs using Adversarial Learning, where pre-trained weight matrices serve as the GAN inputs. Experimental results demonstrate the suitability of the proposed approach under image reconstruction and image classification tasks.
  arXiv  Detail & Related papers  (2021-01-04T16:00:47Z)
• Attentional-Biased Stochastic Gradient Descent [74.49926199036481]
  We present a provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning. Our method is a simple modification to momentum SGD where we assign an individual importance weight to each sample in the mini-batch. ABSGD is flexible enough to combine with other robust losses without any additional cost.
  arXiv  Detail & Related papers  (2020-12-13T03:41:52Z)
• Shaping Deep Feature Space towards Gaussian Mixture for Visual Classification [74.48695037007306]
  We propose a Gaussian mixture (GM) loss function for deep neural networks for visual classification. With a classification margin and a likelihood regularization, the GM loss facilitates both high classification performance and accurate modeling of the feature distribution. The proposed model can be implemented easily and efficiently without using extra trainable parameters.
  arXiv  Detail & Related papers  (2020-11-18T03:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.