X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates

• URL: http://arxiv.org/abs/2509.08729v3
• Date: Wed, 08 Oct 2025 19:54:46 GMT
• Title: X-Teaming Evolutionary M2S: Automated Discovery of Multi-turn to Single-turn Jailbreak Templates
• Abstract summary: Multi-turn-to-single-turn (M2S) compresses iterative red-teaming into one structured prompt, but prior work relied on a handful of manually written templates.<n>We present X-Teaming Evolutionary M2S, an automated framework that discovers and optimize M2S templates through language-model-guided evolution.
• Score: 12.396822247035578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-turn-to-single-turn (M2S) compresses iterative red-teaming into one structured prompt, but prior work relied on a handful of manually written templates. We present X-Teaming Evolutionary M2S, an automated framework that discovers and optimizes M2S templates through language-model-guided evolution. The system pairs smart sampling from 12 sources with an LLM-as-judge inspired by StrongREJECT and records fully auditable logs. Maintaining selection pressure by setting the success threshold to $\theta = 0.70$, we obtain five evolutionary generations, two new template families, and 44.8% overall success (103/230) on GPT-4.1. A balanced cross-model panel of 2,500 trials (judge fixed) shows that structural gains transfer but vary by target; two models score zero at the same threshold. We also find a positive coupling between prompt length and score, motivating length-aware judging. Our results demonstrate that structure-level search is a reproducible route to stronger single-turn probes and underscore the importance of threshold calibration and cross-model evaluation. Code, configurations, and artifacts are available at https://github.com/hyunjun1121/M2S-x-teaming.

Related papers

• MergeSE: Post-Hoc Model Merging for Software Engineering Tasks Without Retraining [9.757638656919832]
  We present textbfMergeSE, an open-source CLI and web tool for training-free model merging of HFace encoder checkpoints.<n> MergeSE supports SE classification more broadly through a built-in registry of nine task types.<n>It supports five merging algorithms, including TIES, DARE-TIES, Wudi, PCB, and averaging.
  arXiv  Detail & Related papers  (2026-08-04T19:39:02Z)
• The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive [0.0]
  CPU-only scoring primitive runs at 2.6 microseconds per token.<n> token rank-frequency distributions converge to the same two- parameter Mandelbrot ranking distribution.<n>We derive a single-pass scoring primitive that composes with model log probabilities when available and degrades to a rank-only mode usable on closed APIs.
  arXiv  Detail & Related papers  (2026-04-28T13:35:31Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• PromptPort: A Reliability Layer for Cross-Model Structured Extraction [0.11280931253550518]
  Structured extraction with LLMs fails in production not because models lack understanding, but because output formatting is unreliable across models and prompts.<n>We present PromptPort, a reliability layer combining deterministic canonicalization with a lightweight verifier (DistilBERT) and a safe-override policy.<n>The method generalizes to held-out model families and provides explicit abstention when uncertain, enabling reliable structured extraction in production deployments.
  arXiv  Detail & Related papers  (2026-01-06T03:54:27Z)
• Cross-LLM Generalization of Behavioral Backdoor Detection in AI Agent Supply Chains [0.0]
  We present the first systematic study of cross-LLM behavioral backdoor detection.<n>We show that single-model detectors achieve 92.7% accuracy within their training distribution but only 49.2% across different LLMs.<n>We show that model-aware detection model identity as an additional feature achieves 90.6% accuracy universally across all evaluated models.
  arXiv  Detail & Related papers  (2025-11-25T03:33:04Z)
• CellARC: Measuring Intelligence with Cellular Automata [0.0]
  CellARC is a synthetic benchmark for abstraction and reasoning built from 1D cellular automata.<n>Each episode has five support pairs and one query serialized in 256 multicolor tokens.<n>We release 95k training episodes plus two 1k test splits (interpolation/extrapolation)
  arXiv  Detail & Related papers  (2025-11-11T07:05:39Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Zero-knowledge LLM hallucination detection and mitigation through fine-grained cross-model consistency [10.052307738781678]
  Large language models (LLMs) have demonstrated impressive capabilities across diverse tasks, but they remain susceptible to hallucinations--generating content that appears plausible but contains factual inaccuracies.<n>We present Finch-Zk, a framework that leverages FINe-grained Cross-model consistency to detect and mitigate Hallucinations in LLM outputs without requiring external knowledge sources.
  arXiv  Detail & Related papers  (2025-08-19T23:45:34Z)
• Evaluating the Use of LLMs for Documentation to Code Traceability [3.076436880934678]
  Large Language Models can establish trace links between various software documentation and source code.<n>We create two novel datasets from two open-source projects (Unity Catalog and Crawl4AI)<n>Results show that the best-performing LLM achieves F1-scores of 79.4% and 80.4% across the two datasets.
  arXiv  Detail & Related papers  (2025-06-19T16:18:53Z)
• APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay [86.01901238059261]
  APIGen-MT is a framework that generates verifiable and diverse multi-turn agent data.<n>We train a family of models -- the xLAM-2-fc-r series with sizes ranging from 1B to 70B parameters.<n>Our models outperform frontier models such as GPT-4o and Claude 3.5 on $tau$-bench and BFCL benchmarks.
  arXiv  Detail & Related papers  (2025-04-04T17:13:57Z)
• Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale [68.6602625868888]
  We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations.<n>Operators in hybrid models can be tailored to token manipulation tasks such as in-context recall, multi-token recall, and compression.<n>We train end-to-end 1.2 to 2.9 times faster than optimized Transformers, and 1.1 to 1.4 times faster than previous generation hybrids.
  arXiv  Detail & Related papers  (2025-02-25T19:47:20Z)
• ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis [50.020850767257095]
  We propose ProofAug, a procedure that equips LLMs with automation methods at various granularities.<n>Our method is validated on the miniF2F benchmark using the open-source deep-math-7b-base model and the Isabelle proof assistant.<n>We also implement a Lean 4 version of ProofAug that can improve the pass@1 performance of Kimina-Prover-seek-Distill-1.5B from 44.3% to 50.4%.
  arXiv  Detail & Related papers  (2025-01-30T12:37:06Z)
• MetaCheckGPT -- A Multi-task Hallucination Detector Using LLM Uncertainty and Meta-models [8.322071110929338]
  This paper describes our winning solution ranked 1st and 2nd in the 2 sub-tasks of model agnostic and model aware tracks respectively. We propose a meta-regressor framework of LLMs for model evaluation and integration that achieves the highest scores on the leaderboard.
  arXiv  Detail & Related papers  (2024-04-10T11:56:01Z)
• Retrosynthesis Prediction with Local Template Retrieval [112.23386062396622]
  Retrosynthesis, which predicts the reactants of a given target molecule, is an essential task for drug discovery. In this work, we introduce RetroKNN, a local reaction template retrieval method. We conduct comprehensive experiments on two widely used benchmarks, the USPTO-50K and USPTO-MIT.
  arXiv  Detail & Related papers  (2023-06-07T03:38:03Z)
• Error Detection in Large-Scale Natural Language Understanding Systems Using Transformer Models [0.0]
  Large-scale conversational assistants like Alexa, Siri, Cortana and Google Assistant process every utterance using multiple models for domain, intent and named entity recognition. We address this challenge to detect domain classification errors using offline Transformer models. We combine utterance encodings from a RoBERTa model with the Nbest hypothesis produced by the production system. We then fine-tune end-to-end in a multitask setting using a small dataset of humanannotated utterances with domain classification errors.
  arXiv  Detail & Related papers  (2021-09-04T00:10:48Z)
• Unsupervised Meta Learning for One Shot Title Compression in Voice Commerce [0.0]
  Product title compression for voice and mobile commerce is a well studied problem with several supervised models proposed so far. We model title compression as a meta learning problem where we ask can we learn a title compression model given only 1 example compression? Our experiments on 16000 crowd generated meta-test examples show that our unsupervised meta training regime is able to acquire a learning algorithm for different tasks after seeing only 1 example for each task.
  arXiv  Detail & Related papers  (2021-02-22T03:53:33Z)
• Meta-Generating Deep Attentive Metric for Few-shot Classification [53.07108067253006]
  We present a novel deep metric meta-generation method to generate a specific metric for a new few-shot learning task. In this study, we structure the metric using a three-layer deep attentive network that is flexible enough to produce a discriminative metric for each task. We gain surprisingly obvious performance improvement over state-of-the-art competitors, especially in the challenging cases.
  arXiv  Detail & Related papers  (2020-12-03T02:07:43Z)
• Robust Finite Mixture Regression for Heterogeneous Targets [70.19798470463378]
  We propose an FMR model that finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously. We provide non-asymptotic oracle performance bounds for our model under a high-dimensional learning framework. The results show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-12T03:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.