Sobolev Training for Operator Learning

• URL: http://arxiv.org/abs/2402.09084v1
• Date: Wed, 14 Feb 2024 10:57:29 GMT
• Title: Sobolev Training for Operator Learning
• Abstract summary: This study investigates the impact of Sobolev Training on operator learning frameworks for improving model performance. Our research reveals that integrating derivative information into the loss function enhances the training process. We propose a novel framework to approximate derivatives on irregular meshes in operator learning.
• Score: 4.97999729336721
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study investigates the impact of Sobolev Training on operator learning frameworks for improving model performance. Our research reveals that integrating derivative information into the loss function enhances the training process, and we propose a novel framework to approximate derivatives on irregular meshes in operator learning. Our findings are supported by both experimental evidence and theoretical analysis. This demonstrates the effectiveness of Sobolev Training in approximating the solution operators between infinite-dimensional spaces.

Related papers

• Sobolev acceleration for neural networks [12.655311590103238]
  Sobolev training has been shown to accelerate convergence and improve generalization compared to conventional $L2$ training.<n>We present the first rigorous theoretical framework proving that Sobolev training accelerates the convergence of Rectified Linear Unit (ReLU) networks.
  arXiv  Detail & Related papers  (2025-09-24T05:52:02Z)
• Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning [50.53703102032562]
  Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks.<n>The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood.
  arXiv  Detail & Related papers  (2025-05-16T08:50:42Z)
• Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
  In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
  arXiv  Detail & Related papers  (2024-10-31T22:54:34Z)
• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning [75.68193159293425]
  In-context learning (ICL) allows transformer-based language models to learn a specific task with a few "task demonstrations" without updating their parameters. We propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
  arXiv  Detail & Related papers  (2024-05-22T15:52:52Z)
• Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models [96.9373147383119]
  We show that weight disentanglement is the crucial factor that makes task arithmetic effective. We show that fine-tuning models in their tangent space by linearizing them amplifies weight disentanglement. This leads to substantial performance improvements across task arithmetic benchmarks and diverse models.
  arXiv  Detail & Related papers  (2023-05-22T08:39:25Z)
• Neural DAEs: Constrained neural networks [4.212663349859165]
  We implement related methods in residual neural networks, despite some fundamental scenario differences. We show when to use which method based on experiments involving simulations of multi-body pendulums and molecular dynamics scenarios. Several of our methods are easy to implement in existing code and have limited impact on training performance while giving significant boosts in terms of inference.
  arXiv  Detail & Related papers  (2022-11-25T18:58:28Z)
• Batch Active Learning from the Perspective of Sparse Approximation [12.51958241746014]
  Active learning enables efficient model training by leveraging interactions between machine learning agents and human annotators. We study and propose a novel framework that formulates batch active learning from the sparse approximation's perspective. Our active learning method aims to find an informative subset from the unlabeled data pool such that the corresponding training loss function approximates its full data pool counterpart.
  arXiv  Detail & Related papers  (2022-11-01T03:20:28Z)
• Task-Free Continual Learning via Online Discrepancy Distance Learning [11.540150938141034]
  This paper develops a new theoretical analysis framework which provides generalization bounds based on the discrepancy distance between the visited samples and the entire information made available for training the model. Inspired by this theoretical model, we propose a new approach enabled by the dynamic component expansion mechanism for a mixture model, namely the Online Discrepancy Distance Learning (ODDL)
  arXiv  Detail & Related papers  (2022-10-12T20:44:09Z)
• A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning [55.048010996144036]
  We show that under some noise assumption, we can obtain the linear spectral feature of its corresponding Markov transition operator in closed-form for free. We propose Spectral Dynamics Embedding (SPEDE), which breaks the trade-off and completes optimistic exploration for representation learning by exploiting the structure of the noise.
  arXiv  Detail & Related papers  (2021-11-22T19:24:57Z)
• Holistic Deep Learning [3.718942345103135]
  This paper presents a novel holistic deep learning framework that addresses the challenges of vulnerability to input perturbations, overparametrization, and performance instability. The proposed framework holistically improves accuracy, robustness, sparsity, and stability over standard deep learning models.
  arXiv  Detail & Related papers  (2021-10-29T14:46:32Z)
• Efficient Estimation of Influence of a Training Instance [56.29080605123304]
  We propose an efficient method for estimating the influence of a training instance on a neural network model. Our method is inspired by dropout, which zero-masks a sub-network and prevents the sub-network from learning each training instance. We demonstrate that the proposed method can capture training influences, enhance the interpretability of error predictions, and cleanse the training dataset for improving generalization.
  arXiv  Detail & Related papers  (2020-12-08T04:31:38Z)
• Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization [22.603803397940133]
  This paper presents a framework for gradient optimization for non functional models such as neural networks. We provide a theoretical analysis method derived from this framework.
  arXiv  Detail & Related papers  (2020-06-30T14:31:24Z)
• On the Benefits of Invariance in Neural Networks [56.362579457990094]
  We show that training with data augmentation leads to better estimates of risk and thereof gradients, and we provide a PAC-Bayes generalization bound for models trained with data augmentation. We also show that compared to data augmentation, feature averaging reduces generalization error when used with convex losses, and tightens PAC-Bayes bounds.
  arXiv  Detail & Related papers  (2020-05-01T02:08:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.