Fine-tuning Factor Augmented Neural Lasso for Heterogeneous Environments

• URL: http://arxiv.org/abs/2604.12288v1
• Date: Tue, 14 Apr 2026 05:01:18 GMT
• Title: Fine-tuning Factor Augmented Neural Lasso for Heterogeneous Environments
• Abstract summary: This paper introduces the fine-tuning factor augmented neural Lasso (FAN-Lasso)<n>FAN-Lasso is a transfer learning framework for high-dimensional nonparametric regression with variable selection.<n>It consistently outperforms standard baselines and achieves near-oracle performance even under severe target sample size constraints.
• Score: 9.523993463071834
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning is a widely used strategy for adapting pre-trained models to new tasks, yet its methodology and theoretical properties in high-dimensional nonparametric settings with variable selection have not yet been developed. This paper introduces the fine-tuning factor augmented neural Lasso (FAN-Lasso), a transfer learning framework for high-dimensional nonparametric regression with variable selection that simultaneously handles covariate and posterior shifts. We use a low-rank factor structure to manage high-dimensional dependent covariates and propose a novel residual fine-tuning decomposition in which the target function is expressed as a transformation of a frozen source function and other variables to achieve transfer learning and nonparametric variable selection. This augmented feature from the source predictor allows for the transfer of knowledge to the target domain and reduces model complexity there. We derive minimax-optimal excess risk bounds for the fine-tuning FAN-Lasso, characterizing the precise conditions, in terms of relative sample sizes and function complexities, under which fine-tuning yields statistical acceleration over single-task learning. The proposed framework also provides a theoretical perspective on parameter-efficient fine-tuning methods. Extensive numerical experiments across diverse covariate- and posterior-shift scenarios demonstrate that the fine-tuning FAN-Lasso consistently outperforms standard baselines and achieves near-oracle performance even under severe target sample size constraints, empirically validating the derived rates.

Related papers

• Advancing Optimal Subset Oracle via Learning Relaxation of Neural Set Functions [41.373161204333435]
  Learning neural set functions is pivotal to a wide range of important applications, including compound selection in AI-driven drug discovery and product recommendation.<n>Recent work has introduced optimal subsets to implicitly learn set functions under practical weakly supervised settings.<n>These frameworks rely on Monte Carlo sampling to estimate gradients of the evidence lower bound when updating the variational distribution.<n>In this work, we reinterpret the evidence lower bound as a continuous relaxation of the set function and learn a surrogate objective that replaces sampling-based ELBO estimation during variational optimization.
  arXiv  Detail & Related papers  (2026-07-13T13:37:47Z)
• Slack More, Predict Better: Proximal Relaxation for Probabilistic Latent Variable Model-based Soft Sensors [12.346122868771118]
  Probabilistic Latent Variable Models (NPLVMs) are a cornerstone of soft sensor modeling.<n>NPLVMs are trained using amortized variational inference, where neural networks parameterize the variational posterior.<n>We introduce KProxNPLVM, a novel NPLVM that pivots to alleviate the objective itself and improving the NPLVM's performance.
  arXiv  Detail & Related papers  (2026-03-12T02:53:39Z)
• Hyperparameter Trajectory Inference with Conditional Lagrangian Optimal Transport [51.56484100374058]
  Post-deployment, user preferences can evolve, making initial settings undesirable.<n>We learn, from observed data, how a NN's conditional output distribution changes with its hyper parameters.<n>We construct a surrogate model that approximates the NN at unobserved hyper parameters.
  arXiv  Detail & Related papers  (2026-03-02T11:55:02Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Variational Entropic Optimal Transport [67.76725267984578]
  We propose Variational Entropic Optimal Transport (VarEOT) for domain translation problems.<n>VarEOT is based on an exact variational reformulation of the log-partition $log mathbbE[exp(cdot)$ as a tractable generalization over an auxiliary positive normalizer.<n> Experiments on synthetic data and unpaired image-to-image translation demonstrate competitive or improved translation quality.
  arXiv  Detail & Related papers  (2026-02-02T15:48:44Z)
• Neural Optimal Transport Meets Multivariate Conformal Prediction [58.43397908730771]
  We propose a framework for conditional vectorile regression (CVQR)<n>CVQR combines neural optimal transport with quantized optimization, and apply it to predictions.
  arXiv  Detail & Related papers  (2025-09-29T19:50:19Z)
• Model-Robust and Adaptive-Optimal Transfer Learning for Tackling Concept Shifts in Nonparametric Regression [7.243632426715939]
  We present a transfer learning procedure that is robust against model misspecification while adaptively attaining optimality.<n>We derive the adaptive convergence rates of the excess risk for specifying Gaussian kernels in a prevalent class of hypothesis transfer learning algorithms.
  arXiv  Detail & Related papers  (2025-01-18T20:33:37Z)
• Causality Pursuit from Heterogeneous Environments via Neural Adversarial Invariance Learning [12.947265104477237]
  Pursuing causality from data is a fundamental problem in scientific discovery, treatment intervention, and transfer learning.<n>This paper introduces a novel algorithmic method for addressing nonparametric invariance and causality learning in regression models across multiple environments.<n>The proposed Focused Adrial Invariant Regularization framework utilizes an innovative minimax optimization approach that drives regression models toward prediction-invariant solutions through adversarial testing.
  arXiv  Detail & Related papers  (2024-05-07T23:37:40Z)
• Scaling and renormalization in high-dimensional regression [72.59731158970894]
  We present a unifying perspective on recent results on ridge regression.<n>We use the basic tools of random matrix theory and free probability, aimed at readers with backgrounds in physics and deep learning.<n>Our results extend and provide a unifying perspective on earlier models of scaling laws.
  arXiv  Detail & Related papers  (2024-05-01T15:59:00Z)
• Neural Control Variates [71.42768823631918]
  We show that a set of neural networks can face the challenge of finding a good approximation of the integrand. We derive a theoretically optimal, variance-minimizing loss function, and propose an alternative, composite loss for stable online training in practice. Specifically, we show that the learned light-field approximation is of sufficient quality for high-order bounces, allowing us to omit the error correction and thereby dramatically reduce the noise at the cost of negligible visible bias.
  arXiv  Detail & Related papers  (2020-06-02T11:17:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.