Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning

• URL: http://arxiv.org/abs/2507.14736v1
• Date: Sat, 19 Jul 2025 19:53:08 GMT
• Title: Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
• Authors: Rafał Surdej, Michał Bortkiewicz, Alex Lewandowski, Mateusz Ostaszewski, Clare Lyle,
• Abstract summary: We study trainable rational activations in both reinforcement and continual learning settings.<n>Our main result is demonstrating a trade-off between expressivity and plasticity in rational activations.<n>Our findings provide actionable design principles for robust and trainable activations in dynamic, nonstationary environments.
• Score: 9.120944934920141
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Trainable activation functions, whose parameters are optimized alongside network weights, offer increased expressivity compared to fixed activation functions. Specifically, trainable activation functions defined as ratios of polynomials (rational functions) have been proposed to enhance plasticity in reinforcement learning. However, their impact on training stability remains unclear. In this work, we study trainable rational activations in both reinforcement and continual learning settings. We find that while their flexibility enhances adaptability, it can also introduce instability, leading to overestimation in RL and feature collapse in longer continual learning scenarios. Our main result is demonstrating a trade-off between expressivity and plasticity in rational activations. To address this, we propose a constrained variant that structurally limits excessive output scaling while preserving adaptability. Experiments across MetaWorld and DeepMind Control Suite (DMC) environments show that our approach improves training stability and performance. In continual learning benchmarks, including MNIST with reshuffled labels and Split CIFAR-100, we reveal how different constraints affect the balance between expressivity and long-term retention. While preliminary experiments in discrete action domains (e.g., Atari) did not show similar instability, this suggests that the trade-off is particularly relevant for continuous control. Together, our findings provide actionable design principles for robust and adaptable trainable activations in dynamic, non-stationary environments. Code available at: https://github.com/special114/rl_rational_plasticity.

Related papers

• Weight Updates as Activation Shifts: A Principled Framework for Steering [54.70188910511715]
  Activation steering promises to be an extremely parameter-efficient form of adaptation, but its effectiveness depends on critical design choices.<n>We establish a first-order equivalence between activation-space interventions and weight-space updates, deriving the conditions under which activation steering can replicate fine-tuning behavior.<n>This equivalence yields a principled framework for steering design and identifies the post-block output as a theoretically-backed and highly expressive intervention site.
  arXiv  Detail & Related papers  (2026-02-28T02:50:04Z)
• Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steering [50.63386303357225]
  We propose AdaRAS, a lightweight test-time framework that improves reasoning reliability by selectively intervening on neuron activations.<n>AdaRAS identifies Reasoning-Critical Neurons (RCNs) via a polarity-aware mean-difference criterion and adaptively steers their activations during inference.<n> Experiments on 10 mathematics and coding benchmarks demonstrate consistent improvements, including over 13% gains on AIME-24 and AIME-25.
  arXiv  Detail & Related papers  (2026-01-27T17:53:01Z)
• FlexAct: Why Learn when you can Pick? [39.92969675794945]
  We introduce a novel framework that employs the Gumbel-Softmax trick to enable discrete yet differentiable selection.<n>Our method dynamically learns the optimal activation function independently of the input.<n>Experiments on synthetic datasets show that our model consistently selects the most suitable activation function.
  arXiv  Detail & Related papers  (2026-01-10T05:51:25Z)
• MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation [16.96094045628127]
  Existing approaches restrict students to following a single golden rationale and treat different reasoning paths independently.<n>This misalignment leads to a degeneration of the student's latent reasoning distribution, causing suboptimal performance.<n>We propose MIND, a capability-filtered framework that transitions passive mimicry to active cognitive construction.
  arXiv  Detail & Related papers  (2026-01-07T09:08:59Z)
• Steering Vision-Language Pre-trained Models for Incremental Face Presentation Attack Detection [62.89126207012712]
  Face Presentation Attack Detection (PAD) demands incremental learning to combat spoofing tactics and domains.<n>Privacy regulations forbid retaining past data, necessitating rehearsal-free learning (RF-IL)
  arXiv  Detail & Related papers  (2025-12-22T04:30:11Z)
• Activation Function Design Sustains Plasticity in Continual Learning [1.618563064839635]
  In continual learning, models can progressively lose the ability to adapt.<n>We show that activation choice is a primary, architecture-agnostic lever for mitigating plasticity loss.
  arXiv  Detail & Related papers  (2025-09-26T16:41:47Z)
• Stochastic Encodings for Active Feature Acquisition [100.47043816019888]
  Active Feature Acquisition is an instance-wise, sequential decision making problem.<n>The aim is to dynamically select which feature to measure based on current observations, independently for each test instance.<n>Common approaches either use Reinforcement Learning, which experiences training difficulties, or greedily maximize the conditional mutual information of the label and unobserved features, which makes myopic.<n>We introduce a latent variable model, trained in a supervised manner. Acquisitions are made by reasoning about the features across many possible unobserved realizations in a latent space.
  arXiv  Detail & Related papers  (2025-08-03T23:48:46Z)
• Adapting to Fragmented and Evolving Data: A Fisher Information Perspective [0.0]
  FADE is a lightweight framework for robust learning under dynamic environments.<n>It employs a shift-aware regularization mechanism anchored in Fisher information geometry.<n>FADE operates online with fixed memory and no access to target labels.
  arXiv  Detail & Related papers  (2025-07-25T06:50:09Z)
• Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models [45.938663388013445]
  We show that a small set of high-impact activations in the last few layers governs long-form reasoning attributes.<n>By simply amplifying these activations and inserting "wait" tokens, we can invoke the long CoT ability without any training.
  arXiv  Detail & Related papers  (2025-05-23T10:07:18Z)
• Focus On This, Not That! Steering LLMs with Adaptive Feature Specification [48.27684487597968]
  Focus Instruction Tuning (FIT) trains large language models to condition their responses by focusing on specific features whilst ignoring others, leading to different behaviours based on what features are specified.<n>We demonstrate that FIT successfully steers behaviour at inference time; (ii) increases robustness by amplifying core task signals and down-weighting spurious cues; (iii) mitigates social bias by suppressing demographic attributes; and (iv) generalises under distribution shifts and to previously unseen focus features.
  arXiv  Detail & Related papers  (2024-10-30T12:01:48Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• REACT: Revealing Evolutionary Action Consequence Trajectories for Interpretable Reinforcement Learning [7.889696505137217]
  We propose Revealing Evolutionary Action Consequence Trajectories (REACT) to enhance the interpretability of Reinforcement Learning (RL) In contrast to the prevalent practice of RL models based on their optimal behavior learned during training, we posit that considering a range of edge-case trajectories provides a more comprehensive understanding of their inherent behavior. Our results highlight its effectiveness in revealing nuanced aspects of RL models' behavior beyond optimal performance, thereby contributing to improved interpretability.
  arXiv  Detail & Related papers  (2024-04-04T10:56:30Z)
• Transformers with Learnable Activation Functions [63.98696070245065]
  We use Rational Activation Function (RAF) to learn optimal activation functions during training according to input data. RAF opens a new research direction for analyzing and interpreting pre-trained models according to the learned activation functions.
  arXiv  Detail & Related papers  (2022-08-30T09:47:31Z)
• Adaptive Rational Activations to Boost Deep Reinforcement Learning [68.10769262901003]
  We motivate why rationals are suitable for adaptable activation functions and why their inclusion into neural networks is crucial. We demonstrate that equipping popular algorithms with (recurrent-)rational activations leads to consistent improvements on Atari games.
  arXiv  Detail & Related papers  (2021-02-18T14:53:12Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.