Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity games

• URL: http://arxiv.org/abs/2403.17674v1
• Date: Tue, 26 Mar 2024 13:02:46 GMT
• Title: Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity games
• Abstract summary: SecurityBot is a reinforcement learning agent mentored by pre-trained RL agents to support cybersecurity operations. It is supported with a profile module to generated behavior guidelines, a memory module to accumulate local experiences, a reflection module to re-evaluate choices, and an action module to reduce action space. Building on the CybORG experiment framework, our experiences show that SecurityBot demonstrates significant performance improvement compared with LLM or RL standalone.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Integrating LLM and reinforcement learning (RL) agent effectively to achieve complementary performance is critical in high stake tasks like cybersecurity operations. In this study, we introduce SecurityBot, a LLM agent mentored by pre-trained RL agents, to support cybersecurity operations. In particularly, the LLM agent is supported with a profile module to generated behavior guidelines, a memory module to accumulate local experiences, a reflection module to re-evaluate choices, and an action module to reduce action space. Additionally, it adopts the collaboration mechanism to take suggestions from pre-trained RL agents, including a cursor for dynamic suggestion taken, an aggregator for multiple mentors' suggestions ranking and a caller for proactive suggestion asking. Building on the CybORG experiment framework, our experiences show that SecurityBot demonstrates significant performance improvement compared with LLM or RL standalone, achieving the complementary performance in the cybersecurity games.

Related papers

• Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations [0.7090965808928492]
  Large Language Model improves autonomous defensive decision-making in ACO environment.<n>Online policy distillation framework reduces model size by several orders of magnitude.<n>Suggests policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies.
  arXiv  Detail & Related papers  (2026-07-30T20:28:02Z)
• Harnessing LLM Agents with Skill Programs [58.356514745548026]
  We introduce HASP, a new framework that upgrades skills into executable Program Functions (PFs)<n>PFs act as executable guardrails that activate on failure-prone states and modify the next action or inject corrective context.<n> HASP drives substantial gains compared to both training-free and training-based methods on web-search, math reasoning, and coding tasks.
  arXiv  Detail & Related papers  (2026-05-18T01:35:11Z)
• StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning [44.2992619825834]
  General agents have given rise to phenomenal applications such as OpenClaw and Claude Code.<n>Agentic Reinforcement Learning (RL) is emerging as a central post-training paradigm for empowering Large Language Models.<n>We argue that the conventional token-level Markov Decision Process (MDP) should be advanced to a step-level MDP formulation.
  arXiv  Detail & Related papers  (2026-04-20T15:22:39Z)
• Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization [61.641777037967366]
  Proactive large language model (LLM) agents aim to actively plan, query, and interact over multiple turns.<n>Agentic reinforcement learning (RL) has emerged as a promising solution for training such agents in multi-turn settings.<n>We propose BAO, an agentic RL framework that combines behavior enhancement to enrich proactive reasoning and information-gathering capabilities.
  arXiv  Detail & Related papers  (2026-02-11T20:40:43Z)
• ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback [53.2744585868162]
  Monitoring step-level tool invocation behaviors in real time is critical for agent deployment.<n>We first construct TS-Bench, a novel benchmark for step-level tool invocation safety detection in LLM agents.<n>We then develop a guardrail model, TS-Guard, using multi-task reinforcement learning.
  arXiv  Detail & Related papers  (2026-01-15T07:54:32Z)
• rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection [49.74493901036598]
  Large language models (LLMs) are post-trained through reinforcement learning (RL) to evolve into Reasoning Language Models (RLMs)<n>This paper proposes a novel reinforced strategy injection mechanism (rSIM) that enables any LLM to become an RLM by employing a small planner.<n> Experimental results show that rSIM enables Qwen2.5-0.5B to become an RLM and significantly outperform Qwen2.5-14B.
  arXiv  Detail & Related papers  (2025-12-09T06:55:39Z)
• Agent-R1: Training Powerful LLM Agents with End-to-End Reinforcement Learning [45.88626187315028]
  Large Language Models (LLMs) are increasingly being explored for building Agents capable of active environmental interaction (e.g., via tool use) to solve complex problems.<n>This paper first revisits and clarifies Reinforcement Learning methodologies for LLM Agents by systematically extending the Markov Decision Process (MDP) framework.<n> Secondly, we introduce Agent-R1, a modular, flexible, and user-friendly training framework for RL-based LLM Agents, designed for straightforward adaptation across diverse task scenarios and interactive environments.
  arXiv  Detail & Related papers  (2025-11-18T13:03:15Z)
• Large Language Model Integration with Reinforcement Learning to Augment Decision-Making in Autonomous Cyber Operations [0.6591422574318154]
  Reinforcement Learning (RL) has shown great potential for autonomous decision-making in the cybersecurity domain.<n>In this study, we integrate external knowledge in the form of a Large Language Model (LLM) pretrained on cybersecurity data.<n>We demonstrate that our guided agent achieves over 2x higher rewards during early training and converges to a favorable policy approximately 4,500 episodes faster than the baseline.
  arXiv  Detail & Related papers  (2025-08-28T23:52:03Z)
• MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability [106.35604230971396]
  Recent advancements in Agent techniques enable Large Language Models (LLMs) to autonomously utilize tools for retrieval, planning, and reasoning.<n>To further enhance the universal search capability of agents, we propose a novel pre-training framework, MaskSearch.<n>In the pre-training stage, we introduce the Retrieval Augmented Mask Prediction (RAMP) task, where the model learns to leverage search tools to fill masked spans.<n>After that, the model is trained on downstream tasks to achieve further improvement.
  arXiv  Detail & Related papers  (2025-05-26T17:58:50Z)
• Training LLM-Based Agents with Synthetic Self-Reflected Trajectories and Partial Masking [61.61356842567952]
  We propose STeP, a novel method for improving LLM-based agent training.<n>We synthesize self-reflected trajectories that include reflections and corrections of error steps.<n>Experiments demonstrate that our method improves agent performance across three representative tasks.
  arXiv  Detail & Related papers  (2025-05-26T14:11:12Z)
• AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents [54.29555239363013]
  We propose a generic black-box fuzzing framework, AgentVigil, to automatically discover and exploit indirect prompt injection vulnerabilities.<n>We evaluate AgentVigil on two public benchmarks, AgentDojo and VWA-adv, where it achieves 71% and 70% success rates against agents based on o3-mini and GPT-4o.<n>We apply our attacks in real-world environments, successfully misleading agents to navigate to arbitrary URLs, including malicious sites.
  arXiv  Detail & Related papers  (2025-05-09T07:40:17Z)
• CheatAgent: Attacking LLM-Empowered Recommender Systems via LLM Agent [32.958798200220286]
  Large Language Model (LLM)-empowered recommender systems (RecSys) have brought significant advances in personalized user experience. We propose a novel attack framework called CheatAgent by harnessing the human-like capabilities of LLMs. Our method first identifies the insertion position for maximum impact with minimal input modification.
  arXiv  Detail & Related papers  (2025-04-13T05:31:37Z)
• SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks [110.20297293596005]
  Large language model (LLM) agents need to perform multi-turn interactions in real-world tasks. Existing multi-turn RL algorithms for optimizing LLM agents fail to perform effective credit assignment over multiple turns while leveraging the generalization capabilities of LLMs. We propose a novel RL algorithm, SWEET-RL, that uses a carefully designed optimization objective to train a critic model with access to additional training-time information. Our experiments demonstrate that SWEET-RL achieves a 6% absolute improvement in success and win rates on ColBench compared to other state-of-the-art multi-turn RL algorithms.
  arXiv  Detail & Related papers  (2025-03-19T17:55:08Z)
• ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning [54.787341008881036]
  We introduce Reinforced Meta-thinking Agents (ReMA), a novel framework that leverages Multi-Agent Reinforcement Learning (MARL) to elicit meta-thinking behaviors. ReMA decouples the reasoning process into two hierarchical agents: a high-level meta-thinking agent responsible for generating strategic oversight and plans, and a low-level reasoning agent for detailed executions. Experimental results demonstrate that ReMA outperforms single-agent RL baselines on complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-03-12T16:05:31Z)
• Improving Retrospective Language Agents via Joint Policy Gradient Optimization [57.35348425288859]
  RetroAct is a framework that jointly optimize both task-planning and self-reflective evolution capabilities in language agents. We develop a two-stage joint optimization process that integrates imitation learning and reinforcement learning. We conduct extensive experiments across various testing environments, demonstrating RetroAct has substantial improvements in task performance and decision-making processes.
  arXiv  Detail & Related papers  (2025-03-03T12:54:54Z)
• Multi-Agent Collaboration in Incident Response with Large Language Models [0.0]
  Incident response (IR) is a critical aspect of cybersecurity, requiring rapid decision-making and coordinated efforts to address cyberattacks effectively. Leveraging large language models (LLMs) as intelligent agents offers a novel approach to enhancing collaboration and efficiency in IR scenarios. This paper explores the application of LLM-based multi-agent collaboration using the Backdoors & Breaches framework.
  arXiv  Detail & Related papers  (2024-12-01T03:12:26Z)
• LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble [4.41983632543407]
  We present a consistency-guided reward ensemble framework (CoREN) for embodied agent learning. The framework employs an adaptive ensemble oftemporally consistent rewards to derive domain-grounded rewards in the training datasets.
  arXiv  Detail & Related papers  (2024-11-26T06:04:10Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Robust LLM safeguarding via refusal feature adversarial training [15.76605079209956]
  Large language models (LLMs) are vulnerable to adversarial attacks that can elicit harmful responses. We propose Refusal Feature Adrial Training (ReFAT), a novel algorithm that efficiently performs adversarial training. Experiment results show that ReFAT significantly improves the robustness of three popular LLMs against a wide range of adversarial attacks.
  arXiv  Detail & Related papers  (2024-09-30T08:41:39Z)
• Large Language Models for Base Station Siting: Intelligent Deployment based on Prompt or Agent [62.16747639440893]
  Large language models (LLMs) and their associated technologies advance, particularly in the realms of prompt engineering and agent engineering. This approach entails the strategic use of well-crafted prompts to infuse human experience and knowledge into these sophisticated LLMs. This integration represents the future paradigm of artificial intelligence (AI) as a service and AI for more ease.
  arXiv  Detail & Related papers  (2024-08-07T08:43:32Z)
• Simulating Classroom Education with LLM-Empowered Agents [52.62324491261461]
  SimClass is a multi-agent classroom simulation framework involving user participation. We recognize representative class roles and introduce a novel class control mechanism for automatic classroom teaching. We demonstrate that LLMs can simulate traditional classroom interaction patterns effectively while enhancing user's experience.
  arXiv  Detail & Related papers  (2024-06-27T14:51:07Z)
• Toward Optimal LLM Alignments Using Two-Player Games [86.39338084862324]
  In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.
  arXiv  Detail & Related papers  (2024-06-16T15:24:50Z)
• Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization [53.510942601223626]
  Large Language Models (LLMs) exhibit robust problem-solving capabilities for diverse tasks. These task solvers necessitate manually crafted prompts to inform task rules and regulate behaviors. We propose Agent-Pro: an LLM-based Agent with Policy-level Reflection and Optimization.
  arXiv  Detail & Related papers  (2024-02-27T15:09:20Z)
• Offline Training of Language Model Agents with Functions as Learnable Weights [39.88545362699836]
  We present a novel paradigm of training Large Language Models (LLMs) agents without modifying the LLM weights. We develop Agentr that employs the LLM to update agents' functions and devise an agent training algorithm with two strategies, roll-back, and early-stop. With extensive experiments, we showcase that the agent training paradigm could significantly improve the performance of representative LLM agents.
  arXiv  Detail & Related papers  (2024-02-17T18:31:21Z)
• Reinforcement Learning from LLM Feedback to Counteract Goal Misgeneralization [0.0]
  We introduce a method to address goal misgeneralization in reinforcement learning (RL) Goal misgeneralization occurs when an agent retains its capabilities out-of-distribution yet pursues a proxy rather than the intended one. This study demonstrates how the Large Language Model can efficiently supervise RL agents.
  arXiv  Detail & Related papers  (2024-01-14T01:09:48Z)
• Experiential Co-Learning of Software-Developing Agents [83.34027623428096]
  Large language models (LLMs) have brought significant changes to various domains, especially in software development. We introduce Experiential Co-Learning, a novel LLM-agent learning framework. Experiments demonstrate that the framework enables agents to tackle unseen software-developing tasks more effectively.
  arXiv  Detail & Related papers  (2023-12-28T13:50:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.