Deep Learning for Functional Data Analysis with Adaptive Basis Layers
- URL: http://arxiv.org/abs/2106.10414v1
- Date: Sat, 19 Jun 2021 04:05:13 GMT
- Title: Deep Learning for Functional Data Analysis with Adaptive Basis Layers
- Authors: Junwen Yao, Jonas Mueller, Jane-Ling Wang
- Abstract summary: We introduce neural networks that employ a new Basis Layer whose hidden units are each basis functions themselves implemented as a micro neural network.
Our architecture learns to apply parsimonious dimension reduction to functional inputs that focuses only on information relevant to the target rather than irrelevant variation in the input function.
- Score: 11.831982475316641
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Despite their widespread success, the application of deep neural networks to
functional data remains scarce today. The infinite dimensionality of functional
data means standard learning algorithms can be applied only after appropriate
dimension reduction, typically achieved via basis expansions. Currently, these
bases are chosen a priori without the information for the task at hand and thus
may not be effective for the designated task. We instead propose to adaptively
learn these bases in an end-to-end fashion. We introduce neural networks that
employ a new Basis Layer whose hidden units are each basis functions themselves
implemented as a micro neural network. Our architecture learns to apply
parsimonious dimension reduction to functional inputs that focuses only on
information relevant to the target rather than irrelevant variation in the
input function. Across numerous classification/regression tasks with functional
data, our method empirically outperforms other types of neural networks, and we
prove that our approach is statistically consistent with low generalization
error. Code is available at: \url{https://github.com/jwyyy/AdaFNN}.
Related papers
- Nonlinear functional regression by functional deep neural network with
kernel embedding [20.306390874610635]
We propose a functional deep neural network with an efficient and fully data-dependent dimension reduction method.
The architecture of our functional net consists of a kernel embedding step, a projection step, and a deep ReLU neural network for the prediction.
The utilization of smooth kernel embedding enables our functional net to be discretization invariant, efficient, and robust to noisy observations.
arXiv Detail & Related papers (2024-01-05T16:43:39Z) - Functional data learning using convolutional neural networks [0.0]
We show how convolutional neural networks can be used in regression and classification learning problems.
We use a specific but typical architecture of a convolutional neural network to perform all the regression exercises.
The method, although simple, shows high accuracy and is promising for future use in engineering and medical applications.
arXiv Detail & Related papers (2023-10-05T04:46:52Z) - Provable Data Subset Selection For Efficient Neural Network Training [73.34254513162898]
We introduce the first algorithm to construct coresets for emphRBFNNs, i.e., small weighted subsets that approximate the loss of the input data on any radial basis function network.
We then perform empirical evaluations on function approximation and dataset subset selection on popular network architectures and data sets.
arXiv Detail & Related papers (2023-03-09T10:08:34Z) - Globally Optimal Training of Neural Networks with Threshold Activation
Functions [63.03759813952481]
We study weight decay regularized training problems of deep neural networks with threshold activations.
We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
arXiv Detail & Related papers (2023-03-06T18:59:13Z) - Neural networks trained with SGD learn distributions of increasing
complexity [78.30235086565388]
We show that neural networks trained using gradient descent initially classify their inputs using lower-order input statistics.
We then exploit higher-order statistics only later during training.
We discuss the relation of DSB to other simplicity biases and consider its implications for the principle of universality in learning.
arXiv Detail & Related papers (2022-11-21T15:27:22Z) - What Can Be Learnt With Wide Convolutional Neural Networks? [69.55323565255631]
We study infinitely-wide deep CNNs in the kernel regime.
We prove that deep CNNs adapt to the spatial scale of the target function.
We conclude by computing the generalisation error of a deep CNN trained on the output of another deep CNN.
arXiv Detail & Related papers (2022-08-01T17:19:32Z) - Deep Neural Network Classifier for Multi-dimensional Functional Data [4.340040784481499]
We propose a new approach, called as functional deep neural network (FDNN), for classifying multi-dimensional functional data.
Specifically, a deep neural network is trained based on the principle components of the training data which shall be used to predict the class label of a future data function.
arXiv Detail & Related papers (2022-05-17T19:22:48Z) - Neural networks with linear threshold activations: structure and
algorithms [1.795561427808824]
We show that 2 hidden layers are necessary and sufficient to represent any function representable in the class.
We also give precise bounds on the sizes of the neural networks required to represent any function in the class.
We propose a new class of neural networks that we call shortcut linear threshold networks.
arXiv Detail & Related papers (2021-11-15T22:33:52Z) - Rank-R FNN: A Tensor-Based Learning Model for High-Order Data
Classification [69.26747803963907]
Rank-R Feedforward Neural Network (FNN) is a tensor-based nonlinear learning model that imposes Canonical/Polyadic decomposition on its parameters.
First, it handles inputs as multilinear arrays, bypassing the need for vectorization, and can thus fully exploit the structural information along every data dimension.
We establish the universal approximation and learnability properties of Rank-R FNN, and we validate its performance on real-world hyperspectral datasets.
arXiv Detail & Related papers (2021-04-11T16:37:32Z) - The Connection Between Approximation, Depth Separation and Learnability
in Neural Networks [70.55686685872008]
We study the connection between learnability and approximation capacity.
We show that learnability with deep networks of a target function depends on the ability of simpler classes to approximate the target.
arXiv Detail & Related papers (2021-01-31T11:32:30Z) - A Use of Even Activation Functions in Neural Networks [0.35172332086962865]
We propose an alternative approach to integrate existing knowledge or hypotheses of data structure by constructing custom activation functions.
We show that using an even activation function in one of the fully connected layers improves neural network performance.
arXiv Detail & Related papers (2020-11-23T20:33:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.