Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)

• URL: http://arxiv.org/abs/2307.09072v2
• Date: Fri, 8 Dec 2023 19:04:45 GMT
• Title: Real-time Inference and Extrapolation via a Diffusion-inspired Temporal Transformer Operator (DiTTO)
• Authors: Oded Ovadia, Vivek Oommen, Adar Kahana, Ahmad Peyvan, Eli Turkel, George Em Karniadakis
• Abstract summary: We propose an operator learning method to solve time-dependent partial differential equations (PDEs) continuously and with extrapolation in time without any temporal discretization. The proposed method, named Diffusion-inspired Temporal Transformer Operator (DiTTO), is inspired by latent diffusion models and their conditioning mechanism. We demonstrate its extrapolation capability on a climate problem by estimating the temperature around the globe for several years, and also in modeling hypersonic flows around a double-cone.
• Score: 1.5728609542259502
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Extrapolation remains a grand challenge in deep neural networks across all application domains. We propose an operator learning method to solve time-dependent partial differential equations (PDEs) continuously and with extrapolation in time without any temporal discretization. The proposed method, named Diffusion-inspired Temporal Transformer Operator (DiTTO), is inspired by latent diffusion models and their conditioning mechanism, which we use to incorporate the temporal evolution of the PDE, in combination with elements from the transformer architecture to improve its capabilities. Upon training, DiTTO can make inferences in real-time. We demonstrate its extrapolation capability on a climate problem by estimating the temperature around the globe for several years, and also in modeling hypersonic flows around a double-cone. We propose different training strategies involving temporal-bundling and sub-sampling and demonstrate performance improvements for several benchmarks, performing extrapolation for long time intervals as well as zero-shot super-resolution in time.

Related papers

• Trajectory Flow Matching with Applications to Clinical Time Series Modeling [77.58277281319253]
  Trajectory Flow Matching (TFM) trains a Neural SDE in a simulation-free manner, bypassing backpropagation through the dynamics. We demonstrate improved performance on three clinical time series datasets in terms of absolute performance and uncertainty prediction.
  arXiv  Detail & Related papers  (2024-10-28T15:54:50Z)
• TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model [11.281386703572842]
  A family of models have been developed, utilizing a temporal auto-regressive generative Transformer architecture. TimeDiT is a general foundation model for time series that employs a denoising diffusion paradigm instead of temporal auto-regressive generation. Extensive experiments conducted on a varity of tasks such as forecasting, imputation, and anomaly detection, demonstrate the effectiveness of TimeDiT.
  arXiv  Detail & Related papers  (2024-09-03T22:31:57Z)
• Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data [39.41800375686212]
  Diffusion Transformer, the backbone of Sora for video generation, successfully scales the capacity of diffusion models. We make the first theoretical step towards bridging diffusion transformers for capturing spatial-temporal dependencies. We highlight how the spatial-temporal dependencies are captured and affect learning efficiency.
  arXiv  Detail & Related papers  (2024-07-23T02:42:43Z)
• PDETime: Rethinking Long-Term Multivariate Time Series Forecasting from the perspective of partial differential equations [49.80959046861793]
  We present PDETime, a novel LMTF model inspired by the principles of Neural PDE solvers. Our experimentation across seven diversetemporal real-world LMTF datasets reveals that PDETime adapts effectively to the intrinsic nature of the data.
  arXiv  Detail & Related papers  (2024-02-25T17:39:44Z)
• Long-term Wind Power Forecasting with Hierarchical Spatial-Temporal Transformer [112.12271800369741]
  Wind power is attracting increasing attention around the world due to its renewable, pollution-free, and other advantages. Accurate wind power forecasting (WPF) can effectively reduce power fluctuations in power system operations. Existing methods are mainly designed for short-term predictions and lack effective spatial-temporal feature augmentation.
  arXiv  Detail & Related papers  (2023-05-30T04:03:15Z)
• TILDE-Q: A Transformation Invariant Loss Function for Time-Series Forecasting [8.086595073181604]
  Time-series forecasting can address real-world problems across various domains, including energy, weather, traffic, and economy. Time-series forecasting is a well-researched field, predicting complex temporal patterns such as sudden changes in sequential data still poses a challenge with current models. We propose a novel, compact loss function called TILDEQ that considers not only amplitude and phase distortions but also allows models to capture the shape of time-series sequences.
  arXiv  Detail & Related papers  (2022-10-26T21:32:20Z)
• Semi-supervised Learning of Partial Differential Operators and Dynamical Flows [68.77595310155365]
  We present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is able to interpolate and the solutions to any intermediate time.
  arXiv  Detail & Related papers  (2022-07-28T19:59:14Z)
• Learning to Accelerate Partial Differential Equations via Latent Global Evolution [64.72624347511498]
  Latent Evolution of PDEs (LE-PDE) is a simple, fast and scalable method to accelerate the simulation and inverse optimization of PDEs. We introduce new learning objectives to effectively learn such latent dynamics to ensure long-term stability. We demonstrate up to 128x reduction in the dimensions to update, and up to 15x improvement in speed, while achieving competitive accuracy.
  arXiv  Detail & Related papers  (2022-06-15T17:31:24Z)
• DiffPD: Differentiable Projective Dynamics with Contact [65.88720481593118]
  We present DiffPD, an efficient differentiable soft-body simulator with implicit time integration. We evaluate the performance of DiffPD and observe a speedup of 4-19 times compared to the standard Newton's method in various applications.
  arXiv  Detail & Related papers  (2021-01-15T00:13:33Z)
• Deep Switching Auto-Regressive Factorization:Application to Time Series Forecasting [16.934920617960085]
  DSARF approximates high dimensional data by a product variables between time dependent weights and spatially dependent factors. DSARF is different from the state-of-the-art techniques in that it parameterizes the weights in terms of a deep switching vector auto-regressive factorization. Our experiments attest the superior performance of DSARF in terms of long- and short-term prediction error, when compared with the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-10T20:15:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.