Interpolation, Approximation and Controllability of Deep Neural Networks

• URL: http://arxiv.org/abs/2309.06015v1
• Date: Tue, 12 Sep 2023 07:29:47 GMT
• Title: Interpolation, Approximation and Controllability of Deep Neural Networks
• Authors: Jingpu Cheng, Qianxiao Li, Ting Lin, Zuowei Shen
• Abstract summary: We consider two properties that arise from supervised learning, namely universal and universal approximation. We give a characterisation of universal equivalence, showing that it holds for essentially any architecture with non-linearity.
• Score: 18.311411538309425
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate the expressive power of deep residual neural networks idealized as continuous dynamical systems through control theory. Specifically, we consider two properties that arise from supervised learning, namely universal interpolation - the ability to match arbitrary input and target training samples - and the closely related notion of universal approximation - the ability to approximate input-target functional relationships via flow maps. Under the assumption of affine invariance of the control family, we give a characterisation of universal interpolation, showing that it holds for essentially any architecture with non-linearity. Furthermore, we elucidate the relationship between universal interpolation and universal approximation in the context of general control systems, showing that the two properties cannot be deduced from each other. At the same time, we identify conditions on the control family and the target function that ensures the equivalence of the two notions.

Related papers

• Emergence of Globally Attracting Fixed Points in Deep Neural Networks With Nonlinear Activations [24.052411316664017]
  We introduce a theoretical framework for the evolution of the kernel sequence, which measures the similarity between the hidden representation for two different inputs. For nonlinear activations, the kernel sequence converges globally to a unique fixed point, which can correspond to similar representations depending on the activation and network architecture. This work provides new insights into the implicit biases of deep neural networks and how architectural choices influence the evolution of representations across layers.
  arXiv  Detail & Related papers  (2024-10-26T07:10:47Z)
• Inferring Relational Potentials in Interacting Systems [56.498417950856904]
  We propose Neural Interaction Inference with Potentials (NIIP) as an alternative approach to discover such interactions. NIIP assigns low energy to the subset of trajectories which respect the relational constraints observed. It allows trajectory manipulation, such as interchanging interaction types across separately trained models, as well as trajectory forecasting.
  arXiv  Detail & Related papers  (2023-10-23T00:44:17Z)
• Learning Complete Topology-Aware Correlations Between Relations for Inductive Link Prediction [121.65152276851619]
  We show that semantic correlations between relations are inherently edge-level and entity-independent. We propose a novel subgraph-based method, namely TACO, to model Topology-Aware COrrelations between relations. To further exploit the potential of RCN, we propose Complete Common Neighbor induced subgraph.
  arXiv  Detail & Related papers  (2023-09-20T08:11:58Z)
• Entanglement statistics of randomly interacting spins [62.997667081978825]
  Entanglement depends on the underlying topology of the interaction among the qubits. We investigate the entanglement in the ground state of systems comprising two and three qubits with random interactions.
  arXiv  Detail & Related papers  (2023-07-18T23:58:32Z)
• A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks [49.870593940818715]
  We study the infinite-width limit of a type of three-layer NN model whose first layer is random and fixed. Our theory accommodates different scaling choices of the model, resulting in two regimes of the MF limit that demonstrate distinctive behaviors.
  arXiv  Detail & Related papers  (2022-10-28T17:26:27Z)
• Convex Analysis of the Mean Field Langevin Dynamics [49.66486092259375]
  convergence rate analysis of the mean field Langevin dynamics is presented. $p_q$ associated with the dynamics allows us to develop a convergence theory parallel to classical results in convex optimization.
  arXiv  Detail & Related papers  (2022-01-25T17:13:56Z)
• Covariance Decomposition as a Universal Limit on Correlations in Networks [2.9443230571766845]
  We show that in a network satisfying a certain condition, the covariance matrix of any feasible correlation can be decomposed as a summation of positive semidefinite matrices each of whose terms corresponds to a source in the network. Our result is universal in the sense that it holds in any physical theory of correlation in networks, including the classical, quantum and all generalized probabilistic theories.
  arXiv  Detail & Related papers  (2021-03-27T08:26:45Z)
• A Universal Model for Cross Modality Mapping by Relational Reasoning [29.081989993636338]
  Cross modality mapping has attracted growing attention in the computer vision community. We propose a GCN-based Reasoning Network (RR-Net) in which inter and intra relations are efficiently computed. Experiments on three example tasks, i.e., image classification, social recommendation and sound recognition, clearly demonstrate the superiority and universality of our proposed model.
  arXiv  Detail & Related papers  (2021-02-26T08:56:24Z)
• Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory [9.210074587720172]
  We explain the universal approximation capabilities of deep residual neural networks through geometric nonlinear control. Inspired by recent work establishing links between residual networks and control systems, we provide a general sufficient condition for a residual network to have the power of universal approximation.
  arXiv  Detail & Related papers  (2020-07-12T14:53:30Z)
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators [72.62940905965267]
  Invertible neural networks based on coupling flows (CF-INNs) have various machine learning applications such as image synthesis and representation learning. Are CF-INNs universal approximators for invertible functions? We prove a general theorem to show the equivalence of the universality for certain diffeomorphism classes.
  arXiv  Detail & Related papers  (2020-06-20T02:07:37Z)
• Complex networks with tuneable dimensions as a universality playground [0.0]
  We discuss the role of a fundamental network parameter for universality, the spectral dimension. By explicit computation we prove that the spectral dimension for this model can be tuned continuously from $1$ to infinity. We propose our model as a tool to probe universal behaviour on inhomogeneous structures and comment on the possibility that the universal behaviour of correlated models on such networks mimics the one of continuous field theories in fractional Euclidean dimensions.
  arXiv  Detail & Related papers  (2020-06-18T10:56:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.