Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots

• URL: http://arxiv.org/abs/2605.29963v1
• Date: Thu, 28 May 2026 14:02:27 GMT
• Title: Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots
• Abstract summary: Honeypots are decoy systems mimicking real system components designed to defend against cyber attacks.<n>Recently, LLMs increasingly serve as simulation backbones for honeypots.<n>Honeyval is a comprehensive evaluation framework for LLM-powered HTTP honeypots.
• Score: 22.377298744054954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Honeypots are decoy systems mimicking real system components designed to defend against cyber attacks. Recently, LLMs increasingly serve as simulation backbones for honeypots. They enable defenders to construct high-interaction honeypots with low system security risks. However, LLM-powered honeypot development lacks a unified evaluation framework. Most evaluations consist of measuring response similarity on fixed commands, manual testing, or real-world deployment. These methods are often not scalable for development, reproducible across evaluations, representative of practical attacks, or adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots in 16 backend applications, using AI hacking agents as attackers, employing two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. Using Honeyval, we conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while, on average, preserving a running cost advantage against agentic attackers. Further, we experiment with different counter-offensive honeypots configurations, and observe unique trade-offs, such as longer interactions at the cost of increased detection.

Related papers

• LLM-Based Penetration Testing in the Presence of Honeypots [0.08699280339422537]
  Large language model (LLM) agents are increasingly employed for offensive cybersecurity tasks.<n>LLM-driven attackers can reason about heterogeneous artifacts and use the honeypot suspicion to guide target-selection decisions.<n>We present a systematic study of honeypot-aware budget allocation for LLM attack agents.
  arXiv  Detail & Related papers  (2026-09-08T01:09:03Z)
• AdvancedShelLM: A Stateful Multi-Agent LLM Honeypot for SSH Deception [0.3262230127283452]
  We present a new LLM-based honeypot design that uses a multi-agent, multi-LLM architecture to address the limitations of the previous shelLM LLM honeypot.<n>Our honeypot, called AdvancedShelLM, uses two LLM agents, a Manager and a Worker, that better understand the commands while reducing incorrect responses and increasing deception.
  arXiv  Detail & Related papers  (2026-06-26T11:39:05Z)
• Towards Agentic Honeynet Configuration [0.8873265946210988]
  Honeypots are deception systems that emulate vulnerable services to collect threat intelligence.<n>In practise network and computational resources limit the number of honeypots that can be exposed.<n>This work investigates an AI-driven agentic architecture that autonomously manages honeypot exposure in response to ongoing attacks.
  arXiv  Detail & Related papers  (2026-03-14T21:15:49Z)
• VelLMes: A high-interaction AI-based deception framework [0.3499870393443268]
  This paper presents an AI-based deception framework called VelLMes.<n>VelLMes is designed to be attacked by humans, so interactivity and realism are key for its performance.<n>We evaluate the generative capabilities and the deception capabilities.
  arXiv  Detail & Related papers  (2025-10-08T13:00:23Z)
• P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMs [49.908234151374785]
  During fine-tuning, large language models (LLMs) are increasingly vulnerable to data-poisoning backdoor attacks.<n>We propose Poison-to-Poison (P2P), a general and effective backdoor defense algorithm.<n>We show that P2P can neutralize malicious backdoors while preserving task performance.
  arXiv  Detail & Related papers  (2025-10-06T05:45:23Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents [84.96249955105777]
  LLM agents may pose a greater risk if misused, but their robustness remains underexplored.<n>We propose a new benchmark called AgentHarm to facilitate research on LLM agent misuse.<n>We find leading LLMs are surprisingly compliant with malicious agent requests without jailbreaking.
  arXiv  Detail & Related papers  (2024-10-11T17:39:22Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• Optimally Blending Honeypots into Production Networks: Hardness and Algorithms [11.847370655794608]
  Honeypot is an important cyber defense technique that can expose attackers new attacks. In this paper, we initiate a systematic study on characterizing the cybersecurity effectiveness of a new paradigm of deploying honeypots.
  arXiv  Detail & Related papers  (2024-01-12T18:54:51Z)
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks [99.23352758320945]
  We propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks on large language models (LLMs) Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense first randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs.
  arXiv  Detail & Related papers  (2023-10-05T17:01:53Z)
• LLM in the Shell: Generative Honeypots [0.26249027950824505]
  Honeypots are essential tools in cybersecurity for early detection, threat intelligence gathering, and analysis of attacker's behavior. This work introduces shelLM, a dynamic and realistic software honeypot based on Large Language Models that generates Linux-like shell output.
  arXiv  Detail & Related papers  (2023-08-31T22:05:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.