Tensor-Valued Time and Inference Path Optimization in Differential Equation-Based Generative Modeling

• URL: http://arxiv.org/abs/2404.14161v2
• Date: Sat, 25 May 2024 08:10:27 GMT
• Title: Tensor-Valued Time and Inference Path Optimization in Differential Equation-Based Generative Modeling
• Authors: Dohoon Lee, Kyogu Lee,
• Abstract summary: This work introduces, for the first time, a tensor-valued time that expands the conventional scalar-valued time into multiple dimensions. We also propose a novel path optimization problem designed to adaptively determine multidimensional inference trajectories.
• Score: 16.874769609089764
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the field of generative modeling based on differential equations, conventional methods utilize scalar-valued time during both the training and inference phases. This work introduces, for the first time, a tensor-valued time that expands the conventional scalar-valued time into multiple dimensions. Additionally, we propose a novel path optimization problem designed to adaptively determine multidimensional inference trajectories using a predetermined differential equation solver and a fixed number of function evaluations. Our approach leverages the stochastic interpolant framework, simulation dynamics, and adversarial training to optimize the inference pathway. Notably, incorporating tensor-valued time during training improves some models' inference performance, even without path optimization. When the adaptive, multidimensional path derived from our optimization process is employed, further performance gains are achieved despite the fixed solver configurations. The introduction of tensor-valued time not only enhances the efficiency of models but also opens new avenues for exploration in training and inference methodologies, highlighting the potential of adaptive multidimensional paths.

Related papers

• Invertible Solution of Neural Differential Equations for Analysis of Irregularly-Sampled Time Series [4.14360329494344]
  We propose an invertible solution of Neural Differential Equations (NDE)-based method to handle the complexities of irregular and incomplete time series data. Our method suggests the variation of Neural Controlled Differential Equations (Neural CDEs) with Neural Flow, which ensures invertibility while maintaining a lower computational burden. At the core of our approach is an enhanced dual latent states architecture, carefully designed for high precision across various time series tasks.
  arXiv  Detail & Related papers  (2024-01-10T07:51:02Z)
• Analyzing and Enhancing the Backward-Pass Convergence of Unrolled Optimization [50.38518771642365]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. A central challenge in this setting is backpropagation through the solution of an optimization problem, which often lacks a closed form. This paper provides theoretical insights into the backward pass of unrolled optimization, showing that it is equivalent to the solution of a linear system by a particular iterative method. A system called Folded Optimization is proposed to construct more efficient backpropagation rules from unrolled solver implementations.
  arXiv  Detail & Related papers  (2023-12-28T23:15:18Z)
• Neural Dynamical Operator: Continuous Spatial-Temporal Model with Gradient-Based and Derivative-Free Optimization Methods [0.0]
  We present a data-driven modeling framework called neural dynamical operator that is continuous in both space and time. A key feature of the neural dynamical operator is the resolution-invariance with respect to both spatial and temporal discretizations. We show that the proposed model can better predict long-term statistics via the hybrid optimization scheme.
  arXiv  Detail & Related papers  (2023-11-20T14:31:18Z)
• Backpropagation of Unrolled Solvers with Folded Optimization [55.04219793298687]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver. This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating efficiently solvable analytical models of backpropagation.
  arXiv  Detail & Related papers  (2023-01-28T01:50:42Z)
• Semi-supervised Learning of Partial Differential Operators and Dynamical Flows [68.77595310155365]
  We present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is able to interpolate and the solutions to any intermediate time.
  arXiv  Detail & Related papers  (2022-07-28T19:59:14Z)
• A theoretical and empirical study of new adaptive algorithms with additional momentum steps and shifted updates for stochastic non-convex optimization [0.0]
  It is thought that adaptive optimization algorithms represent the key pillar behind the of the Learning field. In this paper we introduce adaptive momentum techniques for different non-smooth objective problems.
  arXiv  Detail & Related papers  (2021-10-16T09:47:57Z)
• Second-Order Neural ODE Optimizer [11.92713188431164]
  We show that a specific continuous-time OC methodology, called Differential Programming, can be adopted to derive backward ODEs for higher-order derivatives at the same O(1) memory cost. The resulting method converges much faster than first-order baselines in wall-clock time. Our framework also enables direct architecture optimization, such as the integration time of Neural ODEs, with second-order feedback policies.
  arXiv  Detail & Related papers  (2021-09-29T02:58:18Z)
• Optimization on manifolds: A symplectic approach [127.54402681305629]
  We propose a dissipative extension of Dirac's theory of constrained Hamiltonian systems as a general framework for solving optimization problems. Our class of (accelerated) algorithms are not only simple and efficient but also applicable to a broad range of contexts.
  arXiv  Detail & Related papers  (2021-07-23T13:43:34Z)
• A Discrete Variational Derivation of Accelerated Methods in Optimization [68.8204255655161]
  We introduce variational which allow us to derive different methods for optimization. We derive two families of optimization methods in one-to-one correspondence. The preservation of symplecticity of autonomous systems occurs here solely on the fibers.
  arXiv  Detail & Related papers  (2021-06-04T20:21:53Z)
• DiffPD: Differentiable Projective Dynamics with Contact [65.88720481593118]
  We present DiffPD, an efficient differentiable soft-body simulator with implicit time integration. We evaluate the performance of DiffPD and observe a speedup of 4-19 times compared to the standard Newton's method in various applications.
  arXiv  Detail & Related papers  (2021-01-15T00:13:33Z)
• Adaptive Sampling of Pareto Frontiers with Binary Constraints Using Regression and Classification [0.0]
  We present a novel adaptive optimization algorithm for black-box multi-objective optimization problems with binary constraints. Our method is based on probabilistic regression and classification models, which act as a surrogate for the optimization goals. We also present a novel ellipsoid truncation method to speed up the expected hypervolume calculation.
  arXiv  Detail & Related papers  (2020-08-27T09:15:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.