Kernel Logistic Regression Learning for High-Capacity Hopfield Networks

• URL: http://arxiv.org/abs/2504.07633v2
• Date: Mon, 14 Apr 2025 00:29:35 GMT
• Title: Kernel Logistic Regression Learning for High-Capacity Hopfield Networks
• Authors: Akira Tamamori,
• Abstract summary: Hebbian learning limits Hopfield network storage capacity (pattern-to-neuron ratio around 0.14)<n>We propose Kernel Logistic Regression (KLR) learning. Unlike linear methods, KLR uses kernels to implicitly map patterns to high-dimensional feature space, enhancing separability.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hebbian learning limits Hopfield network storage capacity (pattern-to-neuron ratio around 0.14). We propose Kernel Logistic Regression (KLR) learning. Unlike linear methods, KLR uses kernels to implicitly map patterns to high-dimensional feature space, enhancing separability. By learning dual variables, KLR dramatically improves storage capacity, achieving perfect recall even when pattern numbers exceed neuron numbers (up to ratio 1.5 shown), and enhances noise robustness. KLR demonstrably outperforms Hebbian and linear logistic regression approaches.

Related papers

• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle [53.239242017802056]
  Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM)<n>However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing and Rollout Silencing.<n>We propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition.
  arXiv  Detail & Related papers  (2025-08-07T17:53:47Z)
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation [50.001816497407475]
  We introduce Mixture-of-Recursions (MoR), a unified framework that combines the two axes of efficiency inside a single Recursive Transformer.<n>MoR reuses a shared stack of layers across recursion steps to achieve parameter efficiency, while lightweight routers enable adaptive token-level thinking.<n>We also propose a KV sharing variant that reuses KV pairs from the first recursion, specifically designed to decrease prefill latency and memory footprint.
  arXiv  Detail & Related papers  (2025-07-14T17:49:00Z)
• Quantitative Attractor Analysis of High-Capacity Kernel Logistic Regression Hopfield Networks [0.0]
  This paper quantitatively analyzes the attractor structures in KLR-trained networks via extensive simulations.<n>We evaluate recall from diverse initial states across wide storage loads (up to 4.0 P/N) and noise levels.<n>Our analysis confirms KLR's superior performance: high capacity (up to 4.0 P/N) and robustness.
  arXiv  Detail & Related papers  (2025-05-02T12:13:23Z)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? [67.30809748319486]
  Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs)<n>This study critically examines the current state of RLVR.<n>We find that the current training setup does not elicit fundamentally new reasoning patterns.
  arXiv  Detail & Related papers  (2025-04-18T17:59:56Z)
• Kernel Ridge Regression for Efficient Learning of High-Capacity Hopfield Networks [0.0]
  We propose Kernel Ridge Regression (KRR) as an efficient kernel-based alternative for learning high-capacity Hopfield networks. KRR utilizes the kernel trick and predicts bipolar states via regression, crucially offering a non-iterative, closed-form solution for learning dual variables. Our results demonstrate that KRR achieves state-of-the-art storage capacity (reaching $beta$=1.5) and noise robustness, comparable to KLR.
  arXiv  Detail & Related papers  (2025-04-17T01:17:28Z)
• Hyperspherical Normalization for Scalable Deep Reinforcement Learning [57.016639036237315]
  SimbaV2 is a novel reinforcement learning architecture designed to stabilize optimization. It scales up effectively with larger models and greater compute, achieving state-of-the-art performance on 57 continuous control tasks.
  arXiv  Detail & Related papers  (2025-02-21T08:17:24Z)
• Logarithmic Regret for Online KL-Regularized Reinforcement Learning [51.113248212150964]
  KL-regularization plays a pivotal role in improving efficiency of RL fine-tuning for large language models.<n>Despite its empirical advantage, the theoretical difference between KL-regularized RL and standard RL remains largely under-explored.<n>We propose an optimistic-based KL-regularized online contextual bandit algorithm, and provide a novel analysis of its regret.
  arXiv  Detail & Related papers  (2025-02-11T11:11:05Z)
• Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving [63.155562267383864]
  Deep reinforcement learning (DRL) has shown remarkable success in complex autonomous driving scenarios. DRL models inevitably bring high memory consumption and computation, which hinders their wide deployment in resource-limited autonomous driving devices. We introduce a novel dynamic structured pruning approach that gradually removes a DRL model's unimportant neurons during the training stage.
  arXiv  Detail & Related papers  (2024-02-07T09:00:30Z)
• Solving Kernel Ridge Regression with Gradient Descent for a Non-Constant Kernel [1.5229257192293204]
  KRR is a generalization of linear ridge regression that is non-linear in the data, but linear in the parameters. We address the effects of changing the kernel during training, something that is investigated in this paper. We show theoretically and empirically that using a decreasing bandwidth, we are able to achieve both zero training error in combination with good generalization, and a double descent behavior.
  arXiv  Detail & Related papers  (2023-11-03T07:43:53Z)
• Reinforcement Learning with Fast and Forgetful Memory [10.087126455388276]
  We introduce Fast and Forgetful Memory, an algorithm-agnostic memory model designed specifically for Reinforcement Learning (RL) Our approach constrains the model search space via strong structural priors inspired by computational psychology. Fast and Forgetful Memory exhibits training speeds two orders of magnitude faster than recurrent neural networks (RNNs)
  arXiv  Detail & Related papers  (2023-10-06T09:56:26Z)
• Speed Limits for Deep Learning [67.69149326107103]
  Recent advancement in thermodynamics allows bounding the speed at which one can go from the initial weight distribution to the final distribution of the fully trained network. We provide analytical expressions for these speed limits for linear and linearizable neural networks. Remarkably, given some plausible scaling assumptions on the NTK spectra and spectral decomposition of the labels -- learning is optimal in a scaling sense.
  arXiv  Detail & Related papers  (2023-07-27T06:59:46Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training [23.536294640280087]
  We propose the nested forward automatic differentiation (Forward-AD) for the element-wise activation function for memory-efficient training. Our evaluation shows that nested Forward-AD reduces the memory footprint up to 1.97x than the baseline model.
  arXiv  Detail & Related papers  (2022-09-22T04:48:48Z)
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning [56.83172249278467]
  We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces. We train and validate our approach directly on the Intel NNP-I chip for inference. We additionally achieve 28-78% speed-up compared to the native NNP-I compiler on all three workloads.
  arXiv  Detail & Related papers  (2020-07-14T18:50:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.