Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

• URL: http://arxiv.org/abs/2607.28826v1
• Date: Thu, 30 Jul 2026 20:28:02 GMT
• Title: Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations
• Abstract summary: Large Language Model improves autonomous defensive decision-making in ACO environment.<n>Online policy distillation framework reduces model size by several orders of magnitude.<n>Suggests policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies.
• Score: 0.7090965808928492
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autonomous Cyber Operations (ACO) are increasingly important for defending enterprise networks as cyber threats continue to evolve in sophistication. ACO applications commonly employ Reinforcement Learning (RL) agents to learn defensive behaviors through interaction with environments. However, RL agents typically require extensive exploration during training, often resulting in unstable behavior and poor initial decision-making before converging toward effective defense strategies. In this work, we investigate the use of a Large Language Model (LLM) to improve autonomous defensive decision-making within an ACO environment. Through prompt engineering rather than fine-tuning, we demonstrate that an 8-billion parameter LLM pretrained on cybersecurity data can outperform a baseline RL agent in a modified CybORG CAGE Challenge 2 environment. We then propose an online policy distillation framework that transfers the LLM's defensive policy into a lightweight RL agent containing only 64,910 parameters, reducing model size by several orders of magnitude while maintaining effective defensive capabilities. This provides a pathway toward operationalizing frontier cybersecurity models within lightweight, deployable agents. To evaluate transferability, we construct CybORG scenarios ranging from 4 to 12 hosts and assess the approach across varying network configurations. We also evaluate teacher-guided RL stabilization strategies and observe that none consistently surpass the optimized teacher policy, suggesting policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies. Our results demonstrate the potential of cybersecurity-focused LLMs as sources of expertise for autonomous cyber defense, while policy distillation provides a practical path toward operationalizing frontier cybersecurity models within efficient, scalable agents.

Related papers

• Enhancing Cloud Network Resilience via a Robust LLM-Empowered Multi-Agent Reinforcement Learning Framework [9.460661802313835]
  CyberOps-Bots is a hierarchical multi-agent reinforcement learning framework empowered by Large Language Models (LLMs)<n> Experiments on real cloud datasets show that, compared to state-of-the-art algorithms, CyberOps-Bots maintains network availability 68.5% higher.<n>We will release our framework to the community, facilitating the advancement of robust and autonomous defense in cloud networks.
  arXiv  Detail & Related papers  (2026-01-12T01:25:41Z)
• Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization [51.12422886183246]
  Large Language Models (LLMs) have developed rapidly in web services, delivering unprecedented capabilities while amplifying societal risks.<n>Existing works tend to focus on either isolated jailbreak attacks or static defenses, neglecting the dynamic interplay between evolving threats and safeguards in real-world web contexts.<n>We propose ACE-Safety, a novel framework that jointly optimize attack and defense models by seamlessly integrating two key innovative procedures.
  arXiv  Detail & Related papers  (2025-11-24T15:23:41Z)
• Large Language Model-Based Reward Design for Deep Reinforcement Learning-Driven Autonomous Cyber Defense [3.2661946789427314]
  We propose a large language model (LLM)-based reward design approach to generate autonomous cyber defense policies.<n>Our results suggest that LLM-guided reward designs can lead to effective defense strategies against diverse adversarial behaviors.
  arXiv  Detail & Related papers  (2025-11-20T15:54:08Z)
• Large Language Model Integration with Reinforcement Learning to Augment Decision-Making in Autonomous Cyber Operations [0.6591422574318154]
  Reinforcement Learning (RL) has shown great potential for autonomous decision-making in the cybersecurity domain.<n>In this study, we integrate external knowledge in the form of a Large Language Model (LLM) pretrained on cybersecurity data.<n>We demonstrate that our guided agent achieves over 2x higher rewards during early training and converges to a favorable policy approximately 4,500 episodes faster than the baseline.
  arXiv  Detail & Related papers  (2025-08-28T23:52:03Z)
• Reinforcement Learning for Decision-Level Interception Prioritization in Drone Swarm Defense [51.736723807086385]
  We present a case study demonstrating the practical advantages of reinforcement learning in addressing this challenge.<n>We introduce a high-fidelity simulation environment that captures realistic operational constraints.<n>Agent learns to coordinate multiple effectors for optimal interception prioritization.<n>We evaluate the learned policy against a handcrafted rule-based baseline across hundreds of simulated attack scenarios.
  arXiv  Detail & Related papers  (2025-08-01T13:55:39Z)
• Unveiling the Black Box: A Multi-Layer Framework for Explaining Reinforcement Learning-Based Cyber Agents [4.239727656979701]
  We propose a unified, multi-layer explainability framework for RL-based attacker agents.<n>At the MDP level, we model cyberattacks as a Partially Observable Markov Decision Processes (POMDPs)<n>At the policy level, we analyse the temporal evolution of Q-values and use Prioritised Experience Replay (PER) to surface critical learning transitions.
  arXiv  Detail & Related papers  (2025-05-16T21:29:55Z)
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning [125.96848846966087]
  Training large language models (LLMs) as interactive agents presents unique challenges.<n>While reinforcement learning has enabled progress in static tasks, multi-turn agent RL training remains underexplored.<n>We propose StarPO, a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents.
  arXiv  Detail & Related papers  (2025-04-24T17:57:08Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• Security of Deep Reinforcement Learning for Autonomous Driving: A Survey [20.703911484411737]
  Reinforcement learning (RL) enables agents to learn optimal behaviors through interaction with their environment.<n>Despite its promise, RL is susceptible to attacks designed to compromise policy learning or to induce erroneous decisions by trained agents.
  arXiv  Detail & Related papers  (2022-12-12T18:50:49Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training [71.7750435554693]
  We show that several state-of-the-art RL agents proposed for power system control are vulnerable to adversarial attacks. Specifically, we use an adversary Markov Decision Process to learn an attack policy, and demonstrate the potency of our attack. We propose to use adversarial training to increase the robustness of RL agent against attacks and avoid infeasible operational decisions.
  arXiv  Detail & Related papers  (2021-10-18T00:50:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.