Beyond Rewards in Reinforcement Learning for Cyber Defence

• URL: http://arxiv.org/abs/2602.04809v2
• Date: Thu, 12 Feb 2026 17:29:01 GMT
• Title: Beyond Rewards in Reinforcement Learning for Cyber Defence
• Authors: Elizabeth Bates, Chris Hicks, Vasilios Mavroudis,
• Abstract summary: We evaluate the impact of reward function structure on learning and policy behavioural characteristics.<n>We use a variety of sparse and dense reward functions, two well-established cyber gyms, a range of network sizes and both policy gradient and value-based RL algorithms.<n>Our results show that sparse rewards, provided they are goal aligned and can be encountered frequently, uniquely offer both enhanced training reliability and more effective cyber defence agents with lower-risk policies.
• Score: 11.066125432647466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent years have seen an explosion of interest in autonomous cyber defence agents trained to defend computer networks using deep reinforcement learning. These agents are typically trained in cyber gym environments using dense, highly engineered reward functions which combine many penalties and incentives for a range of (un)desirable states and costly actions. Dense rewards help alleviate the challenge of exploring complex environments but risk biasing agents towards suboptimal and potentially riskier solutions, a critical issue in complex cyber environments. We thoroughly evaluate the impact of reward function structure on learning and policy behavioural characteristics using a variety of sparse and dense reward functions, two well-established cyber gyms, a range of network sizes, and both policy gradient and value-based RL algorithms. Our evaluation is enabled by a novel ground truth evaluation approach which allows directly comparing between different reward functions, illuminating the nuanced inter-relationships between rewards, action space and the risks of suboptimal policies in cyber environments. Our results show that sparse rewards, provided they are goal aligned and can be encountered frequently, uniquely offer both enhanced training reliability and more effective cyber defence agents with lower-risk policies. Surprisingly, sparse rewards can also yield policies that are better aligned with cyber defender goals and make sparing use of costly defensive actions without explicit reward-based numerical penalties.

Related papers

• Adversarial Reinforcement Learning for Offensive and Defensive Agents in a Simulated Zero-Sum Network Environment [3.572219661521267]
  This paper presents a controlled study of adversarial reinforcement learning in network security through a custom OpenAI Gym environment.<n>The environment captures realistic security trade-offs including background traffic noise, progressive exploitation mechanics, IP-based evasion tactics, honeypot traps, and rate-limiting defenses.
  arXiv  Detail & Related papers  (2025-10-03T05:53:51Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Reinforcement Learning for Decision-Level Interception Prioritization in Drone Swarm Defense [51.736723807086385]
  We present a case study demonstrating the practical advantages of reinforcement learning in addressing this challenge.<n>We introduce a high-fidelity simulation environment that captures realistic operational constraints.<n>Agent learns to coordinate multiple effectors for optimal interception prioritization.<n>We evaluate the learned policy against a handcrafted rule-based baseline across hundreds of simulated attack scenarios.
  arXiv  Detail & Related papers  (2025-08-01T13:55:39Z)
• Less is more? Rewards in RL for Cyber Defence [0.24578723416255752]
  We evaluate whether sparse reward functions might enable training more effective cyber defence agents.<n>Our results show that sparse rewards, particularly positive reinforcement for an uncompromised network state, enable the training of more effective cyber defence agents.
  arXiv  Detail & Related papers  (2025-03-05T07:53:39Z)
• Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense [9.927281246704604]
  We propose a hierarchical Proximal Policy Optimization (PPO) architecture that decomposes the cyber defense task into specific sub-tasks like network investigation and host recovery.<n>Our approach involves training sub-policies for each sub-task using PPO enhanced with cybersecurity domain expertise.<n>These sub-policies are then leveraged by a master defense policy that coordinates their selection to solve complex network defense tasks.
  arXiv  Detail & Related papers  (2024-10-22T18:35:05Z)
• Reward Shaping for Happier Autonomous Cyber Security Agents [0.276240219662896]
  One of the most promising directions uses deep reinforcement learning to train autonomous agents in computer network defense tasks. This work studies the impact of the reward signal that is provided to the agents when training for this task.
  arXiv  Detail & Related papers  (2023-10-20T15:04:42Z)
• Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement Learning [4.796742432333795]
  Team of intelligent agents in computer network defence roles may reveal promising avenues to safeguard cyber and kinetic assets. Agents are evaluated on their ability to jointly mitigate attacker activity in host-based defence scenarios.
  arXiv  Detail & Related papers  (2023-08-25T14:07:50Z)
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS [70.60975663021952]
  We study blackbox adversarial attacks on network classifiers. We argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions. We show that a continual learning approach is required to study attacker-defender dynamics.
  arXiv  Detail & Related papers  (2021-11-23T23:42:16Z)
• Explore and Control with Adversarial Surprise [78.41972292110967]
  Reinforcement learning (RL) provides a framework for learning goal-directed policies given user-specified rewards. We propose a new unsupervised RL technique based on an adversarial game which pits two policies against each other to compete over the amount of surprise an RL agent experiences. We show that our method leads to the emergence of complex skills by exhibiting clear phase transitions.
  arXiv  Detail & Related papers  (2021-07-12T17:58:40Z)
• Disturbing Reinforcement Learning Agents with Corrupted Rewards [62.997667081978825]
  We analyze the effects of different attack strategies based on reward perturbations on reinforcement learning algorithms. We show that smoothly crafting adversarial rewards are able to mislead the learner, and that using low exploration probability values, the policy learned is more robust to corrupt rewards.
  arXiv  Detail & Related papers  (2021-02-12T15:53:48Z)
• Learn2Perturb: an End-to-end Feature Perturbation Learning to Improve Adversarial Robustness [79.47619798416194]
  Learn2Perturb is an end-to-end feature perturbation learning approach for improving the adversarial robustness of deep neural networks. Inspired by the Expectation-Maximization, an alternating back-propagation training algorithm is introduced to train the network and noise parameters consecutively.
  arXiv  Detail & Related papers  (2020-03-02T18:27:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.