Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learning

• URL: http://arxiv.org/abs/2407.06756v1
• Date: Tue, 9 Jul 2024 11:07:41 GMT
• Title: Frequency and Generalisation of Periodic Activation Functions in Reinforcement Learning
• Authors: Augustine N. Mavor-Parker, Matthew J. Sargent, Caswell Barry, Lewis Griffin, Clare Lyle,
• Abstract summary: We show that periodic activations learn low frequency representations and as a result avoid overfitting to bootstrapped targets. We also show that weight decay regularization is able to partially offset the overfitting of periodic activation functions.
• Score: 9.6812227037557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Periodic activation functions, often referred to as learned Fourier features have been widely demonstrated to improve sample efficiency and stability in a variety of deep RL algorithms. Potentially incompatible hypotheses have been made about the source of these improvements. One is that periodic activations learn low frequency representations and as a result avoid overfitting to bootstrapped targets. Another is that periodic activations learn high frequency representations that are more expressive, allowing networks to quickly fit complex value functions. We analyse these claims empirically, finding that periodic representations consistently converge to high frequencies regardless of their initialisation frequency. We also find that while periodic activation functions improve sample efficiency, they exhibit worse generalization on states with added observation noise -- especially when compared to otherwise equivalent networks with ReLU activation functions. Finally, we show that weight decay regularization is able to partially offset the overfitting of periodic activation functions, delivering value functions that learn quickly while also generalizing.

Related papers

• Semi-Periodic Activation for Time Series Classification [1.6631602844999722]
  The study comprehensively analyzes properties, such as bounded, monotonic, nonlinearity, and periodicity, for activation in time series neural networks. We propose a new activation that maximizes the coverage of these properties, called LeakySineLU.
  arXiv  Detail & Related papers  (2024-12-13T06:06:49Z)
• Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning [81.98675881423131]
  This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. We introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors.
  arXiv  Detail & Related papers  (2024-03-12T01:28:00Z)
• Denoising and Extension of Response Functions in the Time Domain [48.52478746418526]
  Response functions of quantum systems describe the response of a system to an external perturbation. In equilibrium and steady-state systems, they correspond to a positive spectral function in the frequency domain.
  arXiv  Detail & Related papers  (2023-09-05T20:26:03Z)
• On the Activation Function Dependence of the Spectral Bias of Neural Networks [0.0]
  We study the phenomenon from the point of view of the spectral bias of neural networks. We provide a theoretical explanation for the spectral bias of ReLU neural networks by leveraging connections with the theory of finite element methods. We show that neural networks with the Hat activation function are trained significantly faster using gradient descent and ADAM.
  arXiv  Detail & Related papers  (2022-08-09T17:40:57Z)
• Functional Regularization for Reinforcement Learning via Learned Fourier Features [98.90474131452588]
  We propose a simple architecture for deep reinforcement learning by embedding inputs into a learned Fourier basis. We show that it improves the sample efficiency of both state-based and image-based RL.
  arXiv  Detail & Related papers  (2021-12-06T18:59:52Z)
• Model based Multi-agent Reinforcement Learning with Tensor Decompositions [52.575433758866936]
  This paper investigates generalisation in state-action space over unexplored state-action pairs by modelling the transition and reward functions as tensors of low CP-rank. Experiments on synthetic MDPs show that using tensor decompositions in a model-based reinforcement learning algorithm can lead to much faster convergence if the true transition and reward functions are indeed of low rank.
  arXiv  Detail & Related papers  (2021-10-27T15:36:25Z)
• Periodic Activation Functions Induce Stationarity [19.689175123261613]
  We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.
  arXiv  Detail & Related papers  (2021-10-26T11:10:37Z)
• Modulated Periodic Activations for Generalizable Local Functional Representations [113.64179351957888]
  We present a new representation that generalizes to multiple instances and achieves state-of-the-art fidelity. Our approach produces general functional representations of images, videos and shapes, and achieves higher reconstruction quality than prior works that are optimized for a single signal.
  arXiv  Detail & Related papers  (2021-04-08T17:59:04Z)
• Continuous Submodular Function Maximization [91.17492610120324]
  Continuous submodularity is a class of functions with a wide spectrum of applications. We identify several applications of continuous submodular optimization, ranging from influence, MAP for inferences to inferences to field field.
  arXiv  Detail & Related papers  (2020-06-24T04:37:31Z)
• Neural Networks Fail to Learn Periodic Functions and How to Fix It [6.230751621285322]
  We prove and demonstrate experimentally that the standard activations functions, such as ReLU, tanh, sigmoid, fail to learn to extrapolate simple periodic functions. We propose a new activation, $x + sin2(x)$, which achieves the desired periodic inductive bias to learn a periodic function. Experimentally, we apply the proposed method to temperature and financial data prediction.
  arXiv  Detail & Related papers  (2020-06-15T07:49:33Z)
• RobustPeriod: Time-Frequency Mining for Robust Multiple Periodicity Detection [36.254037216142244]
  We propose a robust and general framework for multiple periodicity detection. Our algorithm applies maximal overlap discrete wavelet transform to transform the time series into multiple temporal-frequency scales. Experiments on synthetic and real-world datasets show that our algorithm outperforms other popular ones for both single and multiple periodicity detection.
  arXiv  Detail & Related papers  (2020-02-21T20:10:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.