Some Best Practices in Operator Learning

• URL: http://arxiv.org/abs/2412.06686v1
• Date: Mon, 09 Dec 2024 17:28:29 GMT
• Title: Some Best Practices in Operator Learning
• Authors: Dustin Enyeart, Guang Lin,
• Abstract summary: It considers the architectures DeepONets, neural operators and Koopman autoencoders for several differential equations to find robust trends. Some options are considered activation functions, dropouts and weight averaging.
• Score: 6.03891813540831
• License:
• Abstract: Hyperparameters searches are computationally expensive. This paper studies some general choices of hyperparameters and training methods specifically for operator learning. It considers the architectures DeepONets, Fourier neural operators and Koopman autoencoders for several differential equations to find robust trends. Some options considered are activation functions, dropout and stochastic weight averaging.

Related papers

• A Library for Learning Neural Operators [77.16483961863808]
  We present NeuralOperator, an open-source Python library for operator learning. Neural operators generalize neural networks to maps between function spaces instead of finite-dimensional Euclidean spaces. Built on top of PyTorch, NeuralOperator provides all the tools for training and deploying neural operator models.
  arXiv  Detail & Related papers  (2024-12-13T18:49:37Z)
• DeepOSets: Non-Autoregressive In-Context Learning of Supervised Learning Operators [11.913853433712855]
  We introduce DeepSets Operator Networks (DeepOSets), an efficient, non-autoregressive neural network architecture for in-context learning of permutation-invariant operators. DeepOSets combines the operator learning capabilities of Deep Operator Networks (DeepONets) with the set learning capabilities of DeepSets.
  arXiv  Detail & Related papers  (2024-10-11T23:07:19Z)
• Dilated convolution neural operator for multiscale partial differential equations [11.093527996062058]
  We propose the Dilated Convolutional Neural Operator (DCNO) for multiscale partial differential equations. The DCNO architecture effectively captures both high-frequency and low-frequency features while maintaining a low computational cost. We show that DCNO strikes an optimal balance between accuracy and computational cost and offers a promising solution for multiscale operator learning.
  arXiv  Detail & Related papers  (2024-07-16T08:17:02Z)
• PICL: Physics Informed Contrastive Learning for Partial Differential Equations [7.136205674624813]
  We develop a novel contrastive pretraining framework that improves neural operator generalization across multiple governing equations simultaneously. A combination of physics-informed system evolution and latent-space model output are anchored to input data and used in our distance function. We find that physics-informed contrastive pretraining improves accuracy for the Fourier Neural Operator in fixed-future and autoregressive rollout tasks for the 1D and 2D Heat, Burgers', and linear advection equations.
  arXiv  Detail & Related papers  (2024-01-29T17:32:22Z)
• Hyena Neural Operator for Partial Differential Equations [9.438207505148947]
  Recent advances in deep learning have provided a new approach to solving partial differential equations that involves the use of neural operators. This study utilizes a neural operator called Hyena, which employs a long convolutional filter that is parameterized by a multilayer perceptron. Our findings indicate Hyena can serve as an efficient and accurate model for partial learning differential equations solution operator.
  arXiv  Detail & Related papers  (2023-06-28T19:45:45Z)
• Towards Learning Universal Hyperparameter Optimizers with Transformers [57.35920571605559]
  We introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction. Our experiments demonstrate that the OptFormer can imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates.
  arXiv  Detail & Related papers  (2022-05-26T12:51:32Z)
• Neural Operator: Learning Maps Between Function Spaces [75.93843876663128]
  We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations.
  arXiv  Detail & Related papers  (2021-08-19T03:56:49Z)
• How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers [86.36020260204302]
  We propose a new benchmarking protocol to evaluate both end-to-end efficiency and data-addition training efficiency. A human study is conducted to show that our evaluation protocol matches human tuning behavior better than the random search. We then apply the proposed benchmarking framework to 7s and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining.
  arXiv  Detail & Related papers  (2020-10-19T21:46:39Z)
• Fourier Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  We formulate a new neural operator by parameterizing the integral kernel directly in Fourier space. We perform experiments on Burgers' equation, Darcy flow, and Navier-Stokes equation. It is up to three orders of magnitude faster compared to traditional PDE solvers.
  arXiv  Detail & Related papers  (2020-10-18T00:34:21Z)
• OpEvo: An Evolutionary Method for Tensor Operator Optimization [6.273446055072434]
  We propose a novel evolutionary method, OpEvo, which efficiently explores the search spaces of tensor operators. Our comprehensive experiment results show that OpEvo can find the best configuration with the lowest variance and least efforts in the number of trials and wall-clock time.
  arXiv  Detail & Related papers  (2020-06-10T05:33:33Z)
• Learning with Differentiable Perturbed Optimizers [54.351317101356614]
  We propose a systematic method to transform operations into operations that are differentiable and never locally constant. Our approach relies on perturbeds, and can be used readily together with existing solvers. We show how this framework can be connected to a family of losses developed in structured prediction, and give theoretical guarantees for their use in learning tasks.
  arXiv  Detail & Related papers  (2020-02-20T11:11:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.