Multi-Agent Reinforcement Learning for Maritime Operational Technology Cyber Security

• URL: http://arxiv.org/abs/2401.10149v1
• Date: Thu, 18 Jan 2024 17:22:22 GMT
• Title: Multi-Agent Reinforcement Learning for Maritime Operational Technology Cyber Security
• Abstract summary: This paper demonstrates the potential for autonomous cyber defence to be applied on industrial control systems. It introduces a simulation environment, IPMSRL, of a generic Integrated Platform Management System (IPMS) It explores the use of MARL for autonomous cyber defence decision-making on generic maritime based IPMS Operational Technology (OT)
• Score: 0.3958317527488535
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper demonstrates the potential for autonomous cyber defence to be applied on industrial control systems and provides a baseline environment to further explore Multi-Agent Reinforcement Learning's (MARL) application to this problem domain. It introduces a simulation environment, IPMSRL, of a generic Integrated Platform Management System (IPMS) and explores the use of MARL for autonomous cyber defence decision-making on generic maritime based IPMS Operational Technology (OT). OT cyber defensive actions are less mature than they are for Enterprise IT. This is due to the relatively brittle nature of OT infrastructure originating from the use of legacy systems, design-time engineering assumptions, and lack of full-scale modern security controls. There are many obstacles to be tackled across the cyber landscape due to continually increasing cyber-attack sophistication and the limitations of traditional IT-centric cyber defence solutions. Traditional IT controls are rarely deployed on OT infrastructure, and where they are, some threats aren't fully addressed. In our experiments, a shared critic implementation of Multi Agent Proximal Policy Optimisation (MAPPO) outperformed Independent Proximal Policy Optimisation (IPPO). MAPPO reached an optimal policy (episode outcome mean of 1) after 800K timesteps, whereas IPPO was only able to reach an episode outcome mean of 0.966 after one million timesteps. Hyperparameter tuning greatly improved training performance. Across one million timesteps the tuned hyperparameters reached an optimal policy whereas the default hyperparameters only managed to win sporadically, with most simulations resulting in a draw. We tested a real-world constraint, attack detection alert success, and found that when alert success probability is reduced to 0.75 or 0.9, the MARL defenders were still able to win in over 97.5% or 99.5% of episodes, respectively.

Related papers

• Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations [0.7090965808928492]
  Large Language Model improves autonomous defensive decision-making in ACO environment.<n>Online policy distillation framework reduces model size by several orders of magnitude.<n>Suggests policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies.
  arXiv  Detail & Related papers  (2026-07-30T20:28:02Z)
• STRIDE-AI: A Threat Modeling Framework for Generative AI Security Assessment [0.0]
  We present textitSTRIDE-AI, a framework that bridges the gap between high-level risk standards (NIST AI RMF) and technical vulnerability.<n>The framework defines a six-phase assessment lifecycle, introduces a threat modeling adaptation of classical STRIDE for AI systems, and is operationalized through a purpose-built web tool.
  arXiv  Detail & Related papers  (2026-05-16T21:27:23Z)
• Explainable Autonomous Cyber Defense using Adversarial Multi-Agent Reinforcement Learning [2.0391237204597368]
  We present the Causal Multi-Agent Decision Framework (C-MADF) for autonomous cyber defense.<n>C-MADF integrates causal modeling with adversarial dual-policy control.<n>On the real-world CICIoT2023 dataset, C-MADF reduces the false-positive rate from 11.2%, 9.7%, and 8.4% in three cutting-edge literature baselines to 1.8%.
  arXiv  Detail & Related papers  (2026-04-06T05:40:43Z)
• DeepStage: Learning Autonomous Defense Policies Against Multi-Stage APT Campaigns [0.0]
  This paper presents DeepStage, a framework for adaptive, stage-aware defense against Advanced Threats (APTs)<n>The enterprise environment is modeled as a partially observable Markov decision process (POMDP), where host provenance and network telemetry are fused into unified provenance graphs.<n>DeepStage achieves a stage-weighted F1-score of 0.89, outperforming a risk-aware DRL baseline by 21.9%.
  arXiv  Detail & Related papers  (2026-03-17T09:46:11Z)
• Toward Quantitative Modeling of Cybersecurity Risks Due to AI Misuse [50.87630846876635]
  We develop nine detailed cyber risk models.<n>Each model decomposes attacks into steps using the MITRE ATT&CK framework.<n>Individual estimates are aggregated through Monte Carlo simulation.
  arXiv  Detail & Related papers  (2025-12-09T17:54:17Z)
• ANNIE: Be Careful of Your Robots [48.89876809734855]
  We present the first systematic study of adversarial safety attacks on embodied AI systems.<n>We show attack success rates exceeding 50% across all safety categories.<n>Results expose a previously underexplored but highly consequential attack surface in embodied AI systems.
  arXiv  Detail & Related papers  (2025-09-03T15:00:28Z)
• Reinforcement Learning for Decision-Level Interception Prioritization in Drone Swarm Defense [56.47577824219207]
  We present a case study demonstrating the practical advantages of reinforcement learning in addressing this challenge.<n>We introduce a high-fidelity simulation environment that captures realistic operational constraints.<n>Agent learns to coordinate multiple effectors for optimal interception prioritization.<n>We evaluate the learned policy against a handcrafted rule-based baseline across hundreds of simulated attack scenarios.
  arXiv  Detail & Related papers  (2025-08-01T13:55:39Z)
• Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition [101.86739402748995]
  We run the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios.<n>We build the Agent Red Teaming benchmark and evaluate it across 19 state-of-the-art models.<n>Our findings highlight critical and persistent vulnerabilities in today's AI agents.
  arXiv  Detail & Related papers  (2025-07-28T05:13:04Z)
• Leveraging Trustworthy AI for Automotive Security in Multi-Domain Operations: Towards a Responsive Human-AI Multi-Domain Task Force for Cyber Social Security [0.7842667530489607]
  Multi-Domain Operations (MDOs) emphasize cross-domain defense against complex and synergistic threats.<n>Civilian infrastructures like smart cities and Connected Autonomous Vehicles (CAVs) are emerging as primary targets.<n>As dual-use assets, CAVs are vulnerable to Multi-Surface Threats (MSTs), particularly from Adversarial Machine Learning (AML)<n>This study investigates how key hyper parameters in Decision Tree-based ensemble models-Random Forest (RF), Gradient Boosting (GB), and Extreme Gradient Boosting (XGB)-affect the time required for a Black-Box AML attack.
  arXiv  Detail & Related papers  (2025-07-23T11:46:52Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage. We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility. Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Applying Action Masking and Curriculum Learning Techniques to Improve Data Efficiency and Overall Performance in Operational Technology Cyber Security using Reinforcement Learning [0.0]
  This paper extends the use of IPMSRL to enhance realism including the additional dynamics of false positive alerts and alert delay. The training method which resulted in the highest level of performance was a combination of the application of curriculum learning and action masking.
  arXiv  Detail & Related papers  (2024-09-13T16:46:55Z)
• Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents [44.34340798542]
  Large Language Models (LLMs) have shown remarkable capabilities in natural language tasks requiring complex reasoning. Traditional supervised pre-training on static datasets falls short in enabling autonomous agent capabilities. We propose a framework that combines guided Monte Carlo Tree Search (MCTS) search with a self-critique mechanism and iterative fine-tuning on agent interactions.
  arXiv  Detail & Related papers  (2024-08-13T20:52:13Z)
• Training on the Fly: On-device Self-supervised Learning aboard Nano-drones within 20 mW [52.280742520586756]
  Miniaturized cyber-physical systems (CPSes) powered by tiny machine learning (TinyML), such as nano-drones, are becoming an increasingly attractive technology. Simple electronics make these CPSes inexpensive, but strongly limit the computational, memory, and sensing resources available on board. We present a novel on-device fine-tuning approach that relies only on the limited ultra-low power resources available aboard nano-drones.
  arXiv  Detail & Related papers  (2024-08-06T13:11:36Z)
• Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systems [27.316115171846953]
  Large Language Models (LLMs) have shown significant promise in real-world decision-making tasks for embodied AI. LLMs are fine-tuned to leverage their inherent common sense and reasoning abilities while being tailored to specific applications. This fine-tuning process introduces considerable safety and security vulnerabilities, especially in safety-critical cyber-physical systems.
  arXiv  Detail & Related papers  (2024-05-27T17:59:43Z)
• Uniformly Safe RL with Objective Suppression for Multi-Constraint Safety-Critical Applications [73.58451824894568]
  The widely adopted CMDP model constrains the risks in expectation, which makes room for dangerous behaviors in long-tail states. In safety-critical domains, such behaviors could lead to disastrous outcomes. We propose Objective Suppression, a novel method that adaptively suppresses the task reward maximizing objectives according to a safety critic.
  arXiv  Detail & Related papers  (2024-02-23T23:22:06Z)
• Moving Target Defense based Secured Network Slicing System in the O-RAN Architecture [12.360792257414458]
  Artificial intelligence (AI) and machine learning (ML) security threats can even threaten open radio access network (O-RAN) benefits. This paper proposes a novel approach to estimating the optimal number of predefined VNFs for each slice. We also address secure AI/ML methods for dynamic service admission control and power minimization in the O-RAN architecture.
  arXiv  Detail & Related papers  (2023-09-23T18:21:33Z)
• Confidence-Controlled Exploration: Efficient Sparse-Reward Policy Learning for Robot Navigation [72.24964965882783]
  Reinforcement learning (RL) is a promising approach for robotic navigation, allowing robots to learn through trial and error. Real-world robotic tasks often suffer from sparse rewards, leading to inefficient exploration and suboptimal policies. We introduce Confidence-Controlled Exploration (CCE), a novel method that improves sample efficiency in RL-based robotic navigation without modifying the reward function.
  arXiv  Detail & Related papers  (2023-06-09T18:45:15Z)
• Distributed-Training-and-Execution Multi-Agent Reinforcement Learning for Power Control in HetNet [48.96004919910818]
  We propose a multi-agent deep reinforcement learning (MADRL) based power control scheme for the HetNet. To promote cooperation among agents, we develop a penalty-based Q learning (PQL) algorithm for MADRL systems. In this way, an agent's policy can be learned by other agents more easily, resulting in a more efficient collaboration process.
  arXiv  Detail & Related papers  (2022-12-15T17:01:56Z)
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS [70.60975663021952]
  We study blackbox adversarial attacks on network classifiers. We argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions. We show that a continual learning approach is required to study attacker-defender dynamics.
  arXiv  Detail & Related papers  (2021-11-23T23:42:16Z)
• Model-predictive control and reinforcement learning in multi-energy system case studies [0.2810625954925815]
  We present an on-objective and off-policy multi- reinforcement learning (RL) approach against a linear model-predictive-control (LMPC) We show that a twin delayed deep deterministic policy gradient (TD3) RL agent offers potential to match and outperform the perfect foresight LMPC benchmark (101.5%) While in a more complex MES system configuration, the RL agent's performance is generally lower (94.6%), yet still better than the realistic LMPC (88.9%)
  arXiv  Detail & Related papers  (2021-04-20T06:51:50Z)
• Ultra-Reliable Indoor Millimeter Wave Communications using Multiple Artificial Intelligence-Powered Intelligent Surfaces [115.85072043481414]
  We propose a novel framework for guaranteeing ultra-reliable millimeter wave (mmW) communications using multiple artificial intelligence (AI)-enabled reconfigurable intelligent surfaces (RISs) The use of multiple AI-powered RISs allows changing the propagation direction of the signals transmitted from a mmW access point (AP) Two centralized and distributed controllers are proposed to control the policies of the mmW AP and RISs.
  arXiv  Detail & Related papers  (2021-03-31T19:15:49Z)
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations [88.94162416324505]
  A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises. Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions. We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks.
  arXiv  Detail & Related papers  (2020-03-19T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.