Related papers: Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization

Related papers

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations [98.44542103979735]
First-order gradient descent methods, epitomized by gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines.<n>Large-scale model training, stringent privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency.<n>We retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstreams across diverse model architectures and training scenarios.<n>We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research.
arXiv Detail & Related papers (2026-04-14T17:01:36Z)
ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
arXiv Detail & Related papers (2026-02-07T10:19:36Z)
Flows and Diffusions on the Neural Manifold [0.0]
Diffusion and flow-based generative models have achieved remarkable success in domains such as image synthesis, video generation, and natural language modeling.<n>We extend these advances to weight space learning by leveraging recent techniques to incorporate structural priors derived from optimization dynamics.
arXiv Detail & Related papers (2025-07-14T02:26:06Z)
Model Steering: Learning with a Reference Model Improves Generalization Bounds and Scaling Laws [52.10468229008941]
This paper formalizes an emerging learning paradigm that uses a trained model as a reference to guide and enhance the training of a target model through strategic data selection or weighting.<n>We provide theoretical insights into why this approach improves generalization and data efficiency compared to training without a reference model.<n>Building on these insights, we introduce a novel method for Contrastive Language-Image Pretraining with a reference model, termed DRRho-CLIP.
arXiv Detail & Related papers (2025-05-10T16:55:03Z)
Fuzzy Rule-based Differentiable Representation Learning [16.706014479049493]
This paper introduces a novel representation learning method grounded in an interpretable fuzzy rule-based model. It is built upon the Takagi-Sugeno-Kang fuzzy system (TSK-FS) to initially map input data to a high-dimensional fuzzy feature space. A novel differentiable optimization method is proposed for the consequence part learning which can preserve the model's interpretability and transparency.
arXiv Detail & Related papers (2025-03-16T14:00:34Z)
A Neural Network Training Method Based on Distributed PID Control [0.0]
In the previous article, we introduced a neural network framework based on symmetric differential equations. This study proposes an alternative training approach that utilizes differential equation signal propagation instead of chain rule derivation.
arXiv Detail & Related papers (2024-11-18T19:25:26Z)
Training Free Guided Flow Matching with Optimal Control [6.729886762762167]
We present OC-Flow, a training-free framework for guided flow matching using optimal control. We show that OC-Flow achieved superior performance in experiments on text-guided image manipulation, conditional molecule generation, and all-atom peptide design.
arXiv Detail & Related papers (2024-10-23T17:53:11Z)
Sobolev Training for Operator Learning [4.97999729336721]
This study investigates the impact of Sobolev Training on operator learning frameworks for improving model performance. Our research reveals that integrating derivative information into the loss function enhances the training process. We propose a novel framework to approximate derivatives on irregular meshes in operator learning.
arXiv Detail & Related papers (2024-02-14T10:57:29Z)
The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Models [75.33431791218302]
Deep Neural Network Network (DNN) models are used for programming purposes. In this paper we examine the use of convex neural recovery models. We show that all the stationary non-dimensional objective objective can be characterized as the standard a global subsampled convex solvers program. We also show that all the stationary non-dimensional objective objective can be characterized as the standard a global subsampled convex solvers program.
arXiv Detail & Related papers (2023-12-19T23:04:56Z)
Provable Reward-Agnostic Preference-Based Reinforcement Learning [61.39541986848391]
Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories. We propose a theoretical reward-agnostic PbRL framework where exploratory trajectories that enable accurate learning of hidden reward functions are acquired.
arXiv Detail & Related papers (2023-05-29T15:00:09Z)
Towards Compute-Optimal Transfer Learning [82.88829463290041]
We argue that zero-shot structured pruning of pretrained models allows them to increase compute efficiency with minimal reduction in performance. Our results show that pruning convolutional filters of pretrained models can lead to more than 20% performance improvement in low computational regimes.
arXiv Detail & Related papers (2023-04-25T21:49:09Z)
Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training [52.39882976848064]
We design a Generalized Krasnoselskii-Mann (GKM) scheme based on fixed-point iterations as our fundamental ODL module. Under the GKM scheme, a Bilevel Meta Optimization (BMO) algorithmic framework is constructed to solve the optimal training and hyper-training variables together.
arXiv Detail & Related papers (2022-06-16T01:50:25Z)
Planning with Diffusion for Flexible Behavior Synthesis [125.24438991142573]
We consider what it would look like to fold as much of the trajectory optimization pipeline as possible into the modeling problem. The core of our technical approach lies in a diffusion probabilistic model that plans by iteratively denoising trajectories.
arXiv Detail & Related papers (2022-05-20T07:02:03Z)
Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
arXiv Detail & Related papers (2021-06-17T17:26:31Z)
Practical Convex Formulation of Robust One-hidden-layer Neural Network Training [12.71266194474117]
We show that the training of a one-hidden-layer, scalar-output fully-connected ReLULU neural network can be reformulated as a finite-dimensional convex program. We derive a convex optimization approach to efficiently solve the "adversarial training" problem. Our method can be applied to binary classification and regression, and provides an alternative to the current adversarial training methods.
arXiv Detail & Related papers (2021-05-25T22:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.