A tutorial on automatic differentiation with complex numbers

• URL: http://arxiv.org/abs/2409.06752v3
• Date: Tue, 10 Dec 2024 18:34:46 GMT
• Title: A tutorial on automatic differentiation with complex numbers
• Abstract summary: This tutorial is a call to action, for users and developers alike, to take complex values seriously when implementing custom gradient propagation rules.<n>This article surveys forward- and reverse-mode automatic differentiation with complex numbers.
• Score: 6.0735728088312175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic differentiation is everywhere, but there exists only minimal documentation of how it works in complex arithmetic beyond stating "derivatives in $\mathbb{C}^d$" $\cong$ "derivatives in $\mathbb{R}^{2d}$" and, at best, shallow references to Wirtinger calculus. Unfortunately, the equivalence $\mathbb{C}^d \cong \mathbb{R}^{2d}$ becomes insufficient as soon as we need to derive custom gradient rules, e.g., to avoid differentiating "through" expensive linear algebra functions or differential equation simulators. To combat such a lack of documentation, this article surveys forward- and reverse-mode automatic differentiation with complex numbers, covering topics such as Wirtinger derivatives, a modified chain rule, and different gradient conventions while explicitly avoiding holomorphicity and the Cauchy--Riemann equations (which would be far too restrictive). To be precise, we will derive, explain, and implement a complex version of Jacobian-vector and vector-Jacobian products almost entirely with linear algebra without relying on complex analysis or differential geometry. This tutorial is a call to action, for users and developers alike, to take complex values seriously when implementing custom gradient propagation rules -- the manuscript explains how.

Related papers

• Total Variation Rates for Riemannian Flow Matching [8.235086108564998]
  We develop a nonasymptotic Total Variation analysis for RFM samplers.<n>Our key technical ingredient is a differential inequality governing the evolution of TV between two manifold ODE flows.
  arXiv  Detail & Related papers  (2026-02-05T01:06:53Z)
• Automated proving in planar geometry based on the complex number identity method and elimination [0.9668407688201359]
  We improve the complex number identity proving method to a fully automated procedure, based on elimination ideals.<n>Our results are presented in Mathematica, Maple and in a new version of the Giac computer algebra system.
  arXiv  Detail & Related papers  (2025-11-18T18:20:17Z)
• Jet Functors and Weil Algebras in Automatic Differentiation: A Geometric Analysis [0.0]
  We present a geometric formulation of automatic differentiation (AD) using jet bundles and Weil algebras.<n>We derive concise statements on correctness, stability, and complexity.<n>This framework interprets AD theory through the lens of differential geometry.
  arXiv  Detail & Related papers  (2025-10-16T06:25:24Z)
• Surrogate to Poincaré inequalities on manifolds for dimension reduction in nonlinear feature spaces [49.1574468325115]
  We aim to approximate a continuously differentiable function $u:mathbbRd rightarrow mathbbRm$ by a composition of functions $fcirc g$ where $g:mathbbRd rightarrow mathbbRm$, $mleq d$, and $f : mathbbRm rightarrow mathbbRR$.<n>For a fixed $g$, we build $f$ using classical regression methods, involving evaluations
  arXiv  Detail & Related papers  (2025-05-03T12:37:27Z)
• Matrix Calculus (for Machine Learning and Beyond) [0.2647285178819813]
  This course introduces the extension of differential calculus to functions on more general vector spaces. It emphasizes practical computational applications, such as large-scale optimization and machine learning.
  arXiv  Detail & Related papers  (2025-01-07T18:38:35Z)
• Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinations [40.77319247558742]
  We study the computational complexity of learning a target function $f_*:mathbbRdtomathbbR$ with additive structure. We prove that a large subset of $f_*$ can be efficiently learned by gradient training of a two-layer neural network.
  arXiv  Detail & Related papers  (2024-06-17T17:59:17Z)
• Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit [75.4661041626338]
  We study the problem of gradient descent learning of a single-index target function $f_*(boldsymbolx) = textstylesigma_*left(langleboldsymbolx,boldsymbolthetarangleright)$ under isotropic Gaussian data. We prove that a two-layer neural network optimized by an SGD-based algorithm learns $f_*$ of arbitrary link function with a sample and runtime complexity of $n asymp T asymp C(q) cdot d
  arXiv  Detail & Related papers  (2024-06-03T17:56:58Z)
• Tempered Calculus for ML: Application to Hyperbolic Model Embedding [70.61101116794549]
  Most mathematical distortions used in ML are fundamentally integral in nature. In this paper, we unveil a grounded theory and tools which can help improve these distortions to better cope with ML requirements. We show how to apply it to a problem that has recently gained traction in ML: hyperbolic embeddings with a "cheap" and accurate encoding along the hyperbolic vsean scale.
  arXiv  Detail & Related papers  (2024-02-06T17:21:06Z)
• Learning Polynomial Problems with $SL(2,\mathbb{R})$ Equivariance [6.5783892500847205]
  We show that neural networks can effectively solve problems in a data-driven fashion, achieving tenfold speedups while retaining high accuracy. We observe that these learning problems are equivariant to the non-compact group $SL(2,mathbbR)$, which consists of area-preserving linear transformations.
  arXiv  Detail & Related papers  (2023-12-04T18:59:19Z)
• Efficiently Learning One-Hidden-Layer ReLU Networks via Schur Polynomials [50.90125395570797]
  We study the problem of PAC learning a linear combination of $k$ ReLU activations under the standard Gaussian distribution on $mathbbRd$ with respect to the square loss. Our main result is an efficient algorithm for this learning task with sample and computational complexity $(dk/epsilon)O(k)$, whereepsilon>0$ is the target accuracy.
  arXiv  Detail & Related papers  (2023-07-24T14:37:22Z)
• Learning High-Dimensional Nonparametric Differential Equations via Multivariate Occupation Kernel Functions [0.31317409221921133]
  Learning a nonparametric system of ordinary differential equations requires learning $d$ functions of $d$ variables. Explicit formulations scale quadratically in $d$ unless additional knowledge about system properties, such as sparsity and symmetries, is available. We propose a linear approach to learning using the implicit formulation provided by vector-valued Reproducing Kernel Hilbert Spaces.
  arXiv  Detail & Related papers  (2023-06-16T21:49:36Z)
• Manifold Free Riemannian Optimization [4.484251538832438]
  A principled framework for solving optimization problems with a smooth manifold $mathcalM$ is proposed. We use a noiseless sample set of the cost function $(x_i, y_i)in mathcalM times mathbbR$ and the intrinsic dimension of the manifold $mathcalM$.
  arXiv  Detail & Related papers  (2022-09-07T16:19:06Z)
• Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds [71.94111815357064]
  In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. One of the popular tools for finding the low-rank approximations is to use the Riemannian optimization.
  arXiv  Detail & Related papers  (2021-03-27T19:56:00Z)
• High-Dimensional Gaussian Process Inference with Derivatives [90.8033626920884]
  We show that in the low-data regime $ND$, the Gram matrix can be decomposed in a manner that reduces the cost of inference to $mathcalO(N2D + (N2)3)$. We demonstrate this potential in a variety of tasks relevant for machine learning, such as optimization and Hamiltonian Monte Carlo with predictive gradients.
  arXiv  Detail & Related papers  (2021-02-15T13:24:41Z)
• Multiscale regression on unknown manifolds [13.752772802705978]
  We construct low-dimensional coordinates on $mathcalM$ at multiple scales and perform multiscale regression by local fitting. We analyze the generalization error of our method by proving finite sample bounds in high probability on rich classes of priors. Our algorithm has quasilinear complexity in the sample size, with constants linear in $D$ and exponential in $d$.
  arXiv  Detail & Related papers  (2021-01-13T15:14:31Z)
• Automatic Differentiation in ROOT [62.997667081978825]
  In mathematics and computer algebra, automatic differentiation (AD) is a set of techniques to evaluate the derivative of a function specified by a computer program. This paper presents AD techniques available in ROOT, supported by Cling, to produce derivatives of arbitrary C/C++ functions.
  arXiv  Detail & Related papers  (2020-04-09T09:18:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.