Adaptive control of recurrent neural networks using conceptors

• URL: http://arxiv.org/abs/2405.07236v1
• Date: Sun, 12 May 2024 09:58:03 GMT
• Title: Adaptive control of recurrent neural networks using conceptors
• Authors: Guillaume Pourcel, Mirko Goldmann, Ingo Fischer, Miguel C. Soriano,
• Abstract summary: Recurrent Neural Networks excel at predicting and generating complex high-dimensional temporal patterns. In a Machine Learning setting, the network's parameters are adapted during a training phase to match the requirements of a given task/problem. We demonstrate how keeping parts of the network adaptive even after the training enhances its functionality and robustness.
• Score: 1.9686770963118383
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recurrent Neural Networks excel at predicting and generating complex high-dimensional temporal patterns. Due to their inherent nonlinear dynamics and memory, they can learn unbounded temporal dependencies from data. In a Machine Learning setting, the network's parameters are adapted during a training phase to match the requirements of a given task/problem increasing its computational capabilities. After the training, the network parameters are kept fixed to exploit the learned computations. The static parameters thereby render the network unadaptive to changing conditions, such as external or internal perturbation. In this manuscript, we demonstrate how keeping parts of the network adaptive even after the training enhances its functionality and robustness. Here, we utilize the conceptor framework and conceptualize an adaptive control loop analyzing the network's behavior continuously and adjusting its time-varying internal representation to follow a desired target. We demonstrate how the added adaptivity of the network supports the computational functionality in three distinct tasks: interpolation of temporal patterns, stabilization against partial network degradation, and robustness against input distortion. Our results highlight the potential of adaptive networks in machine learning beyond training, enabling them to not only learn complex patterns but also dynamically adjust to changing environments, ultimately broadening their applicability.

Related papers

• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Contrastive Representation Learning for Dynamic Link Prediction in Temporal Networks [1.9389881806157312]
  We introduce a self-supervised method for learning representations of temporal networks. We propose a recurrent message-passing neural network architecture for modeling the information flow over time-respecting paths of temporal networks. The proposed method is tested on Enron, COLAB, and Facebook datasets.
  arXiv  Detail & Related papers  (2024-08-22T22:50:46Z)
• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• A simple theory for training response of deep neural networks [0.0]
  Deep neural networks give us a powerful method to model the training dataset's relationship between input and output. We show the training response consists of some different factors based on training stages, activation functions, or training methods. In addition, we show feature space reduction as an effect of training dynamics, which can result in network fragility.
  arXiv  Detail & Related papers  (2024-05-07T07:20:15Z)
• Dynamical stability and chaos in artificial neural network trajectories along training [3.379574469735166]
  We study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network. We find hints of regular and chaotic behavior depending on the learning rate regime. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning.
  arXiv  Detail & Related papers  (2024-04-08T17:33:11Z)
• Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control [63.310780486820796]
  We show how a parameterization of recurrent connectivity influences robustness in closed-loop settings. We find that closed-form continuous-time neural networks (CfCs) with fewer parameters can outperform their full-rank, fully-connected counterparts.
  arXiv  Detail & Related papers  (2023-10-05T21:44:18Z)
• Dynamic Encoding and Decoding of Information for Split Learning in Mobile-Edge Computing: Leveraging Information Bottleneck Theory [1.1151919978983582]
  Split learning is a privacy-preserving distributed learning paradigm in which an ML model is split into two parts (i.e., an encoder and a decoder) In mobile-edge computing, network functions can be trained via split learning where an encoder resides in a user equipment (UE) and a decoder resides in the edge network. We present a new framework and training mechanism to enable a dynamic balancing of the transmission resource consumption with the informativeness of the shared latent representations.
  arXiv  Detail & Related papers  (2023-09-06T07:04:37Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Learning Fast and Slow for Online Time Series Forecasting [76.50127663309604]
  Fast and Slow learning Networks (FSNet) is a holistic framework for online time-series forecasting. FSNet balances fast adaptation to recent changes and retrieving similar old knowledge. Our code will be made publicly available.
  arXiv  Detail & Related papers  (2022-02-23T18:23:07Z)
• Continual Adaptation for Deep Stereo [52.181067640300014]
  We propose a continual adaptation paradigm for deep stereo networks designed to deal with challenging and ever-changing environments. In our paradigm, the learning signals needed to continuously adapt models online can be sourced from self-supervision via right-to-left image warping or from traditional stereo algorithms. Our network architecture and adaptation algorithms realize the first real-time self-adaptive deep stereo system.
  arXiv  Detail & Related papers  (2020-07-10T08:15:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.