Large Language Model Integration with Reinforcement Learning to Augment Decision-Making in Autonomous Cyber Operations

• URL: http://arxiv.org/abs/2509.05311v1
• Date: Thu, 28 Aug 2025 23:52:03 GMT
• Title: Large Language Model Integration with Reinforcement Learning to Augment Decision-Making in Autonomous Cyber Operations
• Abstract summary: Reinforcement Learning (RL) has shown great potential for autonomous decision-making in the cybersecurity domain.<n>In this study, we integrate external knowledge in the form of a Large Language Model (LLM) pretrained on cybersecurity data.<n>We demonstrate that our guided agent achieves over 2x higher rewards during early training and converges to a favorable policy approximately 4,500 episodes faster than the baseline.
• Score: 0.6591422574318154
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) has shown great potential for autonomous decision-making in the cybersecurity domain, enabling agents to learn through direct environment interaction. However, RL agents in Autonomous Cyber Operations (ACO) typically learn from scratch, requiring them to execute undesirable actions to learn their consequences. In this study, we integrate external knowledge in the form of a Large Language Model (LLM) pretrained on cybersecurity data that our RL agent can directly leverage to make informed decisions. By guiding initial training with an LLM, we improve baseline performance and reduce the need for exploratory actions with obviously negative outcomes. We evaluate our LLM-integrated approach in a simulated cybersecurity environment, and demonstrate that our guided agent achieves over 2x higher rewards during early training and converges to a favorable policy approximately 4,500 episodes faster than the baseline.

Related papers

• Proving the Utility of Large Language Models in Cybersecurity Simulations: A Comprehensive Examination [4.208276864195654]
  This paper explores how Large Language Models (LLMs) can bolster cybersecurity simulations.<n>We employ YAML as a structured representation format for simulating complex network configurations.<n> Comparative studies examine the advantages of LLM-based techniques over classical approaches.
  arXiv  Detail & Related papers  (2026-08-17T11:22:20Z)
• Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations [0.7090965808928492]
  Large Language Model improves autonomous defensive decision-making in ACO environment.<n>Online policy distillation framework reduces model size by several orders of magnitude.<n>Suggests policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies.
  arXiv  Detail & Related papers  (2026-07-30T20:28:02Z)
• Adapting Generalist Robot Policies with Semantic Reinforcement Learning [59.009122725071386]
  Generalist robot policies learn a diverse repertoire of behaviors from large-scale pretraining.<n>We propose Semantic Action Reinforcement Learning (SARL) to learn to optimize this prompt space through online interaction.<n>We show SARL unlocks fundamentally new capabilities -- adapting VLA behavior to solve complex, long-horizon tasks.
  arXiv  Detail & Related papers  (2026-06-30T17:00:33Z)
• LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity [13.20232770906974]
  Large language model (LLM) agents are rapidly being integrated into real-world systems.<n>This survey focuses on the role of LLM agents in empowering the cybersecurity lifecycle across offense and defense.
  arXiv  Detail & Related papers  (2026-06-26T09:32:11Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning [73.35191368656224]
  We present textitEmbodied Reasoning Agent (ERA), a framework that integrates prior knowledge learning and online reinforcement learning.<n>ERA offers a practical path toward scalable embodied intelligence, providing methodological insights for future embodied AI systems.
  arXiv  Detail & Related papers  (2025-10-14T16:25:46Z)
• The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback [51.144727949988436]
  Reinforcement learning (RL) has demonstrated potential to enhance the reasoning capabilities of large language models (LLMs)<n>In this work, we explore improving LLMs through RL with minimal data.<n>To minimize data dependency, we introduce two novel mechanisms grounded in self-awareness.
  arXiv  Detail & Related papers  (2025-10-03T06:32:10Z)
• CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning [1.3863707631653515]
  Cyber defenders are overwhelmed by a deluge of security alerts, threat intelligence signals, and shifting business context.<n>Existing evaluations do not fully assess the scenarios most relevant to real-world defenders.<n>We introduce CyberSOCEval, a new suite of open source benchmarks within CyberSecEval 4.
  arXiv  Detail & Related papers  (2025-09-24T14:33:07Z)
• LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation [7.054214377609925]
  Reinforcement learning (RL) has achieved notable success in various domains.<n>Training effective policies for complex tasks remains challenging.<n>Existing approaches to mitigate training bottlenecks fall into two categories.
  arXiv  Detail & Related papers  (2025-05-27T03:40:02Z)
• Automatic Curriculum Learning for Driving Scenarios: Towards Robust and Efficient Reinforcement Learning [11.602831593017427]
  This paper addresses the challenges of training end-to-end autonomous driving agents using Reinforcement Learning (RL)<n>RL agents are typically trained in a fixed set of scenarios and nominal behavior of surrounding road users in simulations.<n>We propose an automatic curriculum learning framework that dynamically generates driving scenarios with adaptive complexity based on the agent's evolving capabilities.
  arXiv  Detail & Related papers  (2025-05-13T06:26:57Z)
• WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model [55.276852838877346]
  Self-evolving agents are trained on trajectories sampled autonomously based on their own policies.<n>We propose a novel framework that introduces a co-evolving World Model LLM.<n>This world model predicts the next observation based on the current observation and action within the web environment.
  arXiv  Detail & Related papers  (2025-04-23T02:54:31Z)
• Reinforcement Learning Environment with LLM-Controlled Adversary in D&D 5th Edition Combat [0.0]
  This research employs Deep Q-Networks (DQN) for the smaller agents, creating a testbed for strategic AI development.<n>We successfully integrated sophisticated language models into the RL framework, enhancing strategic decision-making processes.
  arXiv  Detail & Related papers  (2025-03-19T22:48:20Z)
• OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilities [0.0]
  We demonstrate a new approach to assessing AI's progress towards enabling and scaling real-world offensive cyber operations.<n>We detail OCCULT, a lightweight operational evaluation framework that allows cyber security experts to contribute to rigorous and repeatable measurement.<n>We find that there has been significant recent advancement in the risks of AI being used to scale realistic cyber threats.
  arXiv  Detail & Related papers  (2025-02-18T19:33:14Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process.<n>We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• On the Modeling Capabilities of Large Language Models for Sequential Decision Making [52.128546842746246]
  Large pretrained models are showing increasingly better performance in reasoning and planning tasks. We evaluate their ability to produce decision-making policies, either directly, by generating actions, or indirectly. In environments with unfamiliar dynamics, we explore how fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities.
  arXiv  Detail & Related papers  (2024-10-08T03:12:57Z)
• Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents [101.17919953243107]
  GovSim is a generative simulation platform designed to study strategic interactions and cooperative decision-making in large language models (LLMs)<n>We find that all but the most powerful LLM agents fail to achieve a sustainable equilibrium in GovSim, with the highest survival rate below 54%.<n>We show that agents that leverage "Universalization"-based reasoning, a theory of moral thinking, are able to achieve significantly better sustainability.
  arXiv  Detail & Related papers  (2024-04-25T15:59:16Z)
• Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity games [0.0]
  SecurityBot is a reinforcement learning agent mentored by pre-trained RL agents to support cybersecurity operations. It is supported with a profile module to generated behavior guidelines, a memory module to accumulate local experiences, a reflection module to re-evaluate choices, and an action module to reduce action space. Building on the CybORG experiment framework, our experiences show that SecurityBot demonstrates significant performance improvement compared with LLM or RL standalone.
  arXiv  Detail & Related papers  (2024-03-26T13:02:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.