Towards Theoretical Understanding of Flexible Transmitter Networks via Approximation and Local Minima

• URL: http://arxiv.org/abs/2111.06027v1
• Date: Thu, 11 Nov 2021 02:41:23 GMT
• Title: Towards Theoretical Understanding of Flexible Transmitter Networks via Approximation and Local Minima
• Authors: Jin-Hui Wu, Shao-Qun Zhang, Yuan Jiang, Zhi-Hua Zhou
• Abstract summary: We study the theoretical properties of one-hidden-layer FTNet from the perspectives of approximation and local minima. Our results indicate that FTNet can efficiently express target functions and has no concern about local minima.
• Score: 74.30120779041428
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Flexible Transmitter Network (FTNet) is a recently proposed bio-plausible neural network and has achieved competitive performance with the state-of-the-art models when handling temporal-spatial data. However, there remains an open problem about the theoretical understanding of FTNet. This work investigates the theoretical properties of one-hidden-layer FTNet from the perspectives of approximation and local minima. Under mild assumptions, we show that: i) FTNet is a universal approximator; ii) the approximation complexity of FTNet can be exponentially smaller than those of real-valued neural networks with feedforward/recurrent architectures and is of the same order in the worst case; iii) any local minimum of FTNet is the global minimum, which suggests that it is possible for local search algorithms to converge to the global minimum. Our theoretical results indicate that FTNet can efficiently express target functions and has no concern about local minima, which complements the theoretical blank of FTNet and exhibits the possibility for ameliorating the FTNet.

Related papers

• Communication-Efficient Federated Learning by Quantized Variance Reduction for Heterogeneous Wireless Edge Networks [55.467288506826755]
  Federated learning (FL) has been recognized as a viable solution for local-privacy-aware collaborative model training in wireless edge networks. Most existing communication-efficient FL algorithms fail to reduce the significant inter-device variance. We propose a novel communication-efficient FL algorithm, named FedQVR, which relies on a sophisticated variance-reduced scheme.
  arXiv  Detail & Related papers  (2025-01-20T04:26:21Z)
• Learning Load Balancing with GNN in MPTCP-Enabled Heterogeneous Networks [13.178956651532213]
  We propose a graph neural network (GNN)-based model to tackle the LB problem for MP TCP-enabled HetNets. Compared to the conventional deep neural network (DNN), the proposed GNN-based model exhibits two key strengths.
  arXiv  Detail & Related papers  (2024-10-22T15:49:53Z)
• Graph Neural Networks for Power Allocation in Wireless Networks with Full Duplex Nodes [10.150768420975155]
  Due to mutual interference between users, power allocation problems in wireless networks are often non-trivial. Graph Graph neural networks (GNNs) have recently emerged as a promising approach tackling these problems and an approach exploits underlying topology of wireless networks.
  arXiv  Detail & Related papers  (2023-03-27T10:59:09Z)
• Universal Neural Optimal Transport [0.0]
  UNOT (Universal Neural Optimal Transport) is a novel framework capable of accurately predicting (entropic) OT distances and plans. We show that our network not only accurately predicts optimal transport distances and plans across a wide range of datasets, but also captures the geometry of the Wasserstein space correctly.
  arXiv  Detail & Related papers  (2022-11-30T21:56:09Z)
• Towards Understanding Theoretical Advantages of Complex-Reaction Networks [77.34726150561087]
  We show that a class of functions can be approximated by a complex-reaction network using the number of parameters. For empirical risk minimization, our theoretical result shows that the critical point set of complex-reaction networks is a proper subset of that of real-valued networks.
  arXiv  Detail & Related papers  (2021-08-15T10:13:49Z)
• Fast Fourier Intrinsic Network [41.95712986029093]
  We propose the Fast Fourier Intrinsic Network, FFI-Net, that operates in the spectral domain. Weights in FFI-Net are optimized in the spectral domain, allowing faster convergence to a lower error. It achieves state-of-the-art performance on MPI-Sintel, MIT Intrinsic, and IIW datasets.
  arXiv  Detail & Related papers  (2020-11-09T18:14:39Z)
• A non-causal FFTNet architecture for speech enhancement [18.583426581177278]
  We suggest a new parallel, non-causal and shallow waveform domain architecture for speech enhancement based on FFTNet. By suggesting a shallow network and applying non-causality within certain limits, the suggested FFTNet uses much fewer parameters compared to other neural network based approaches.
  arXiv  Detail & Related papers  (2020-06-08T10:49:04Z)
• Flexible Transmitter Network [84.90891046882213]
  Current neural networks are mostly built upon the MP model, which usually formulates the neuron as executing an activation function on the real-valued weighted aggregation of signals received from other neurons. We propose the Flexible Transmitter (FT) model, a novel bio-plausible neuron model with flexible synaptic plasticity. We present the Flexible Transmitter Network (FTNet), which is built on the most common fully-connected feed-forward architecture.
  arXiv  Detail & Related papers  (2020-04-08T06:55:12Z)
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
  This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-06T15:51:00Z)
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions [76.05981545084738]
  We propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts. We show that the proposed ReActNet outperforms all the state-of-the-arts by a large margin.
  arXiv  Detail & Related papers  (2020-03-07T02:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.