Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations

• URL: http://arxiv.org/abs/2602.05523v1
• Date: Thu, 05 Feb 2026 10:30:57 GMT
• Title: Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations
• Abstract summary: Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag benchmarks.<n>We introduce CTF challenge families, whereby a single CTF is used as the basis for generating a family of semantically-equivalent challenges.<n>We introduce a new tool, Evolve-CTF, that generates CTF families from Python challenges using a range of transformations.
• Score: 9.234598988803407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag (CTF) benchmarks. However, existing pointwise benchmarks have limited ability to shed light on the robustness and generalisation abilities of agents across alternative versions of the source code. We introduce CTF challenge families, whereby a single CTF is used as the basis for generating a family of semantically-equivalent challenges via semantics-preserving program transformations. This enables controlled evaluation of agent robustness to source code transformations while keeping the underlying exploit strategy fixed. We introduce a new tool, Evolve-CTF, that generates CTF families from Python challenges using a range of transformations. Using Evolve-CTF to derive families from Cybench and Intercode challenges, we evaluate 13 agentic LLM configurations with tool access. We find that models are remarkably robust to intrusive renaming and code insertion-based transformations, but that composed transformations and deeper obfuscation affect performance by requiring more sophisticated use of tools. We also find that enabling explicit reasoning has little effect on solution success rates across challenge families. Our work contributes a valuable technique and tool for future LLM evaluations, and a large dataset characterising the capabilities of current state-of-the-art models in this domain.

Related papers

• Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach [12.79484512803839]
  Large language model (LLM) agents are increasingly extended through Agent Skills.<n>As open-source skill marketplaces expand, users and agents increasingly rely on brief metadata to select third-party skills.<n>We propose Progressive Loading-Aware Hierarchical Contrastive Learning (PL-HCL) to detect cross-layer misalignment.
  arXiv  Detail & Related papers  (2026-07-12T02:07:37Z)
• CTFusion: A CTF-based Benchmark for LLM Agent Evaluation [15.091299960425857]
  We present CTFusion, a streaming evaluation framework built on Live CTFs.<n>We demonstrate that existing CTF benchmarks can be unreliable in assessing LLM-based agents.<n>We release CTFusion as open source to foster future research in this area.
  arXiv  Detail & Related papers  (2026-05-12T04:23:42Z)
• FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments [60.3427704389541]
  Large Language Models are being increasingly deployed as the decision-making core of autonomous agents.<n>Yet, in conversational benchmarks, these agents frequently fail due to the cascading effects of incorrect decision-making.<n>We present the Failure-Aware Meta-Agentic framework to tackle these challenges.
  arXiv  Detail & Related papers  (2026-04-28T02:21:53Z)
• Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data [52.80567461420316]
  We propose a Memory-Augmented LLM-based Multi-Agent System (textbfMALMAS) for automated feature generation.<n>MALMAS decomposes the generation process into agents with distinct responsibilities, and a Router Agent activates an appropriate subset of agents per iteration.<n>We further integrate a memory module comprising procedural memory, feedback memory, and conceptual memory, enabling iterative refinement that adaptively guides subsequent feature generation.
  arXiv  Detail & Related papers  (2026-04-22T07:09:30Z)
• STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving [1.266832699428407]
  We introduce STRIATUM-CTF (A Search-based Test-time Reasoning Inference Agent for Tactical Utility Maximization in Cybersecurity), a modular agentic framework built upon the Model Context Protocol (MCP)<n>By standardizing tool interfaces for system introspection, decompilation, and runtime debug, STRIATUM-CTF enables the agent to maintain a coherent context window across extended exploit trajectories.<n>Our system participated in a university-hosted Capture-the-Flag (CTF) competition in late 2025, where it operated autonomously to identify and exploit vulnerabilities in real-time.
  arXiv  Detail & Related papers  (2026-03-23T21:17:26Z)
• Fool Me If You Can: On the Robustness of Binary Code Similarity Detection Models against Semantics-preserving Transformations [7.222996408214315]
  We evaluate the robustness of deep learning models for the task of binary code similarity detection.<n>We construct a dataset of 9,565 binary variants from 620 baseline samples.
  arXiv  Detail & Related papers  (2026-02-13T07:23:15Z)
• Evolving Demonstration Optimization for Chain-of-Thought Feature Transformation [25.08795071237411]
  Feature Transformation is a core data-centric AI task that improves feature space quality to advance predictive performance.<n>Existing solutions rely on discrete search or latent generation, but they are frequently limited by sample inefficiency, invalid candidates, and redundant generations with limited coverage.<n>We propose a framework that optimize context data for LLM-driven FT by evolving trajectory-level experiences in a closed loop.
  arXiv  Detail & Related papers  (2026-02-13T01:12:41Z)
• Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMs [69.28193153685893]
  Large Language Models (LLMs) frequently exhibit strong translation abilities, even without task-specific fine-tuning.<n>To demystify this process, we leverage Sparse Autoencoders (SAEs) and introduce a novel framework for identifying task-specific features.<n>Our work not only decodes a core component of the translation mechanism in LLMs but also provides a blueprint for using internal model mechanism to create more robust and efficient models.
  arXiv  Detail & Related papers  (2026-01-16T06:29:07Z)
• Attribution-Guided Decoding [24.52258081219335]
  We introduce Attribution-Guided Decoding (AGD), an interpretability-based decoding strategy.<n>Instead of directly manipulating model activations, AGD considers a set of high-probability output token candidates.<n>We demonstrate AGD's efficacy across three challenging domains.
  arXiv  Detail & Related papers  (2025-09-30T14:21:40Z)
• Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges [0.0]
  'Random-Crypto' is a procedurally generated cryptographic dataset designed to unlock the potential of Reinforcement Learning.<n>We fine-tune a Python tool-augmented Llama-3.1-8B via Group Relative Policy Optimization.<n>The resulting agent achieves a significant improvement in Pass@8 on previously unseen challenges.
  arXiv  Detail & Related papers  (2025-06-01T01:59:52Z)
• Leveraging LLM Inconsistency to Boost Pass@k Performance [3.797421474324735]
  Large language models (LLMs) achieve impressive abilities in numerous domains, but exhibit inconsistent performance in response to minor input changes.<n>We introduce a novel method for leveraging models' inconsistency to boost Pass@k performance.<n>Specifically, we present a "Variator" agent that generates k variants of a given task and submits one candidate solution for each one.
  arXiv  Detail & Related papers  (2025-05-19T10:22:04Z)
• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use.<n>MeCo quantifies metacognitive scores by capturing high-level cognitive signals in the representation space.<n>MeCo is fine-tuning-free and incurs minimal cost.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities [46.34031902647788]
  We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges.<n>We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities.<n> Empirical analysis on 390 CTF challenges demonstrate that these new tools and interfaces substantially improve our agent's performance.
  arXiv  Detail & Related papers  (2024-09-24T15:06:01Z)
• CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning [17.614980614656407]
  We propose Continual Generative training for Incremental prompt-Learning. We exploit Variational Autoencoders to learn class-conditioned distributions. We show that such a generative replay approach can adapt to new tasks while improving zero-shot capabilities.
  arXiv  Detail & Related papers  (2024-07-22T16:51:28Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)
• FLAT: Few-Shot Learning via Autoencoding Transformation Regularizers [67.46036826589467]
  We present a novel regularization mechanism by learning the change of feature representations induced by a distribution of transformations without using the labels of data examples. It could minimize the risk of overfitting into base categories by inspecting the transformation-augmented variations at the encoded feature level. Experiment results show the superior performances to the current state-of-the-art methods in literature.
  arXiv  Detail & Related papers  (2019-12-29T15:26:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.