Kernel Ridge Regression for Efficient Learning of High-Capacity Hopfield Networks

• URL: http://arxiv.org/abs/2504.12561v3
• Date: Fri, 02 May 2025 05:06:56 GMT
• Title: Kernel Ridge Regression for Efficient Learning of High-Capacity Hopfield Networks
• Authors: Akira Tamamori,
• Abstract summary: We propose Kernel Ridge Regression (KRR) as an efficient kernel-based alternative for learning high-capacity Hopfield networks.<n>KRR utilizes the kernel trick and predicts bipolar states via regression, crucially offering a non-iterative, closed-form solution for learning dual variables.<n>Our results demonstrate that KRR achieves state-of-the-art storage capacity (reaching $beta$=1.5) and noise robustness, comparable to KLR.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hopfield networks using Hebbian learning suffer from limited storage capacity. While supervised methods like Linear Logistic Regression (LLR) offer some improvement, kernel methods like Kernel Logistic Regression (KLR) significantly enhance capacity and noise robustness. However, KLR requires computationally expensive iterative learning. We propose Kernel Ridge Regression (KRR) as an efficient kernel-based alternative for learning high-capacity Hopfield networks. KRR utilizes the kernel trick and predicts bipolar states via regression, crucially offering a non-iterative, closed-form solution for learning dual variables. We evaluate KRR and compare its performance against Hebbian, LLR, and KLR. Our results demonstrate that KRR achieves state-of-the-art storage capacity (reaching $\beta$=1.5) and noise robustness, comparable to KLR. Crucially, KRR drastically reduces training time, being orders of magnitude faster than LLR and significantly faster than KLR, especially at higher storage loads. This establishes KRR as a potent and highly efficient method for building high-performance associative memories, providing comparable performance to KLR with substantial training speed advantages. This work provides the first empirical comparison between KRR and KLR in the context of Hopfield network learning.

Related papers

• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle [53.239242017802056]
  Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM)<n>However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing and Rollout Silencing.<n>We propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition.
  arXiv  Detail & Related papers  (2025-08-07T17:53:47Z)
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation [50.001816497407475]
  We introduce Mixture-of-Recursions (MoR), a unified framework that combines the two axes of efficiency inside a single Recursive Transformer.<n>MoR reuses a shared stack of layers across recursion steps to achieve parameter efficiency, while lightweight routers enable adaptive token-level thinking.<n>We also propose a KV sharing variant that reuses KV pairs from the first recursion, specifically designed to decrease prefill latency and memory footprint.
  arXiv  Detail & Related papers  (2025-07-14T17:49:00Z)
• KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning [72.53466291156604]
  We present textbfKDRL, a textitunified post-training framework that jointly optimize a reasoning model through teacher supervision (KD) and self-exploration (RL)<n>We first formulate a unified objective that integrates GRPO and KD, and systematically explore how different KL approximations, KL coefficients, and reward-guided KD strategies affect the overall post-training dynamics and performance.
  arXiv  Detail & Related papers  (2025-06-02T19:46:41Z)
• Reinforced Latent Reasoning for LLM-based Recommendation [83.18146814163308]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks.<n>Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data.<n>In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning.
  arXiv  Detail & Related papers  (2025-05-25T11:03:45Z)
• Joker: Joint Optimization Framework for Lightweight Kernel Machines [20.45405359815043]
  We propose Joker, a joint optimization framework for diverse kernel models, including KRR, logistic regression, and support vector machines.<n> Experiments show that Joker saves up to 90% memory but achieves comparable training time and performance (or even better) than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-23T11:36:45Z)
• Quantitative Attractor Analysis of High-Capacity Kernel Logistic Regression Hopfield Networks [0.0]
  This paper quantitatively analyzes the attractor structures in KLR-trained networks via extensive simulations.<n>We evaluate recall from diverse initial states across wide storage loads (up to 4.0 P/N) and noise levels.<n>Our analysis confirms KLR's superior performance: high capacity (up to 4.0 P/N) and robustness.
  arXiv  Detail & Related papers  (2025-05-02T12:13:23Z)
• Kernel Logistic Regression Learning for High-Capacity Hopfield Networks [0.0]
  Hebbian learning limits Hopfield network storage capacity (pattern-to-neuron ratio around 0.14)<n>We propose Kernel Logistic Regression (KLR) learning. Unlike linear methods, KLR uses kernels to implicitly map patterns to high-dimensional feature space, enhancing separability.
  arXiv  Detail & Related papers  (2025-04-10T10:27:43Z)
• Logarithmic Regret for Online KL-Regularized Reinforcement Learning [51.113248212150964]
  KL-regularization plays a pivotal role in improving efficiency of RL fine-tuning for large language models.<n>Despite its empirical advantage, the theoretical difference between KL-regularized RL and standard RL remains largely under-explored.<n>We propose an optimistic-based KL-regularized online contextual bandit algorithm, and provide a novel analysis of its regret.
  arXiv  Detail & Related papers  (2025-02-11T11:11:05Z)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning [73.93639228235622]
  Continual Learning with foundation models has emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks. Existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks. We propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal.
  arXiv  Detail & Related papers  (2025-01-22T20:00:41Z)
• Dynamic Learning Rate for Deep Reinforcement Learning: A Bandit Approach [0.9549646359252346]
  In deep Reinforcement Learning (RL) models trained using gradient-based techniques, the choice of gradient and its learning rate are crucial to achieving good performance. We propose dynamic Learning Rate for deep Reinforcement Learning (LRRL), a meta-learning approach that selects the learning rate based on the agent's performance during training.
  arXiv  Detail & Related papers  (2024-10-16T14:15:28Z)
• Scaling Optimal LR Across Token Horizons [81.29631219839311]
  We show how optimal learning rate depends on token horizon in LLM training. We also provide evidence that LLama-1 used too high LR, and estimate the performance hit from this.
  arXiv  Detail & Related papers  (2024-09-30T03:32:02Z)
• Have ASkotch: A Neat Solution for Large-scale Kernel Ridge Regression [16.836685923503868]
  ASkotch is a scalable, accelerated, iterative method for full KRR that provably obtains linear convergence. ASkotch outperforms state-of-the-art KRR solvers on a testbed of 23 large-scale KRR regression and classification tasks. Our work opens up the possibility of as-yet-unimagined applications of full KRR across a number of disciplines.
  arXiv  Detail & Related papers  (2024-07-14T04:11:10Z)
• Compressing Deep Reinforcement Learning Networks with a Dynamic Structured Pruning Method for Autonomous Driving [63.155562267383864]
  Deep reinforcement learning (DRL) has shown remarkable success in complex autonomous driving scenarios. DRL models inevitably bring high memory consumption and computation, which hinders their wide deployment in resource-limited autonomous driving devices. We introduce a novel dynamic structured pruning approach that gradually removes a DRL model's unimportant neurons during the training stage.
  arXiv  Detail & Related papers  (2024-02-07T09:00:30Z)
• Solving Kernel Ridge Regression with Gradient Descent for a Non-Constant Kernel [1.5229257192293204]
  KRR is a generalization of linear ridge regression that is non-linear in the data, but linear in the parameters. We address the effects of changing the kernel during training, something that is investigated in this paper. We show theoretically and empirically that using a decreasing bandwidth, we are able to achieve both zero training error in combination with good generalization, and a double descent behavior.
  arXiv  Detail & Related papers  (2023-11-03T07:43:53Z)
• Reinforcement Learning with Fast and Forgetful Memory [10.087126455388276]
  We introduce Fast and Forgetful Memory, an algorithm-agnostic memory model designed specifically for Reinforcement Learning (RL) Our approach constrains the model search space via strong structural priors inspired by computational psychology. Fast and Forgetful Memory exhibits training speeds two orders of magnitude faster than recurrent neural networks (RNNs)
  arXiv  Detail & Related papers  (2023-10-06T09:56:26Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)
• SreaMRAK a Streaming Multi-Resolution Adaptive Kernel Algorithm [60.61943386819384]
  Existing implementations of KRR require that all the data is stored in the main memory. We propose StreaMRAK - a streaming version of KRR. We present a showcase study on two synthetic problems and the prediction of the trajectory of a double pendulum.
  arXiv  Detail & Related papers  (2021-08-23T21:03:09Z)
• Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning [56.17667147101263]
  In real-world tasks, reinforcement learning agents encounter situations that are not present during training time. To ensure reliable performance, the RL agents need to exhibit robustness against worst-case situations. We propose the Robust Hallucinated Upper-Confidence RL (RH-UCRL) algorithm to provably solve this problem.
  arXiv  Detail & Related papers  (2021-03-18T16:50:17Z)
• MLR-SNet: Transferable LR Schedules for Heterogeneous Tasks [56.66010634895913]
  The learning rate (LR) is one of the most important hyper-learned network parameters in gradient descent (SGD) training networks (DNN) In this paper, we propose to learn a proper LR schedule for MLR-SNet tasks. We also make MLR-SNet to query tasks like different noises, architectures, data modalities, sizes from the training ones, and achieve or even better performance.
  arXiv  Detail & Related papers  (2020-07-29T01:18:58Z)
• Towards Understanding Label Smoothing [36.54164997035046]
  Label smoothing regularization (LSR) has a great success in deep neural networks by training algorithms. We show that an appropriate LSR can help to speed up convergence by reducing the variance. We propose a simple yet effective strategy, namely Two-Stage LAbel smoothing algorithm (TSLA)
  arXiv  Detail & Related papers  (2020-06-20T20:36:17Z)
• kDecay: Just adding k-decay items on Learning-Rate Schedule to improve Neural Networks [5.541389959719384]
  k-decay is effectively improves the performance of commonly used and easy LR schedule. We evaluate the k-decay method on CIFAR And ImageNet datasets with different neural networks. The accuracy has been improved by 1.08% on the CIFAR-10 dataset and by 2.07% on the CIFAR-100 dataset.
  arXiv  Detail & Related papers  (2020-04-13T12:58:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.