Adaptive Group Lasso Neural Network Models for Functions of Few Variables and Time-Dependent Data

• URL: http://arxiv.org/abs/2108.10825v1
• Date: Tue, 24 Aug 2021 16:16:46 GMT
• Title: Adaptive Group Lasso Neural Network Models for Functions of Few Variables and Time-Dependent Data
• Authors: Lam Si Tung Ho and Giang Tran
• Abstract summary: We approximate the target function by a deep neural network and enforce an adaptive group Lasso constraint to the weights of a suitable hidden layer. Our empirical studies show that the proposed method outperforms recent state-of-the-art methods including the sparse dictionary matrix method.
• Score: 4.18804572788063
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose an adaptive group Lasso deep neural network for high-dimensional function approximation where input data are generated from a dynamical system and the target function depends on few active variables or few linear combinations of variables. We approximate the target function by a deep neural network and enforce an adaptive group Lasso constraint to the weights of a suitable hidden layer in order to represent the constraint on the target function. Our empirical studies show that the proposed method outperforms recent state-of-the-art methods including the sparse dictionary matrix method, neural networks with or without group Lasso penalty.

Related papers

• Continual Learning via Sequential Function-Space Variational Inference [65.96686740015902]
  We propose an objective derived by formulating continual learning as sequential function-space variational inference. Compared to objectives that directly regularize neural network predictions, the proposed objective allows for more flexible variational distributions. We demonstrate that, across a range of task sequences, neural networks trained via sequential function-space variational inference achieve better predictive accuracy than networks trained with related methods.
  arXiv  Detail & Related papers  (2023-12-28T18:44:32Z)
• Sparse-Input Neural Network using Group Concave Regularization [10.103025766129006]
  Simultaneous feature selection and non-linear function estimation are challenging in neural networks. We propose a framework of sparse-input neural networks using group concave regularization for feature selection in both low-dimensional and high-dimensional settings.
  arXiv  Detail & Related papers  (2023-07-01T13:47:09Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Learning k-Level Structured Sparse Neural Networks Using Group Envelope Regularization [4.0554893636822]
  We introduce a novel approach to deploy large-scale Deep Neural Networks on constrained resources. The method speeds up inference time and aims to reduce memory demand and power consumption.
  arXiv  Detail & Related papers  (2022-12-25T15:40:05Z)
• Learning to Learn with Generative Models of Neural Network Checkpoints [71.06722933442956]
  We construct a dataset of neural network checkpoints and train a generative model on the parameters. We find that our approach successfully generates parameters for a wide range of loss prompts. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.
  arXiv  Detail & Related papers  (2022-09-26T17:59:58Z)
• Level set learning with pseudo-reversible neural networks for nonlinear dimension reduction in function approximation [8.28646586439284]
  We propose a new method of Dimension Reduction via Learning Level Sets (DRiLLS) for function approximation. Our method contains two major components: one is the pseudo-reversible neural network (PRNN) module that effectively transforms high-dimensional input variables to low-dimensional active variables. The PRNN not only relaxes the invertibility constraint of the nonlinear transformation present in the NLL method due to the use of RevNet, but also adaptively weights the influence of each sample and controls the sensitivity the function to the learned active variables.
  arXiv  Detail & Related papers  (2021-12-02T17:25:34Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• Consistent Feature Selection for Analytic Deep Neural Networks [3.42658286826597]
  We investigate the problem of feature selection for analytic deep networks. We prove that for a wide class of networks, the Adaptive Group Lasso selection procedure with Group Lasso is selection-consistent. The work provides further evidence that Group Lasso might be inefficient for feature selection with neural networks.
  arXiv  Detail & Related papers  (2020-10-16T01:59:53Z)
• Model Fusion with Kullback--Leibler Divergence [58.20269014662046]
  We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors.
  arXiv  Detail & Related papers  (2020-07-13T03:27:45Z)
• Linearly Constrained Neural Networks [0.5735035463793007]
  We present a novel approach to modelling and learning vector fields from physical systems using neural networks. To achieve this, the target function is modelled as a linear transformation of an underlying potential field, which is in turn modelled by a neural network.
  arXiv  Detail & Related papers  (2020-02-05T01:27:29Z)
• Model Fusion via Optimal Transport [64.13185244219353]
  We present a layer-wise model fusion algorithm for neural networks. We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
  arXiv  Detail & Related papers  (2019-10-12T22:07:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.