FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs

• URL: http://arxiv.org/abs/2609.02168v1
• Date: Wed, 02 Sep 2026 06:31:39 GMT
• Title: FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs
• Abstract summary: Fragmented safety evaluation undermines the governance of dangerous AI capabilities.<n>We present a modular framework that evaluates each model through three orthogonal pipelines---Knowledge, Defense, and Harm.<n>Our first contribution is a horizontal comparison of dangerous capability across models and model families.<n>The second is a temporal analysis of capability evolution: tracking $K$, $D$, and $H$ against model release dates reveals that dangerous capability has not monotonically declined.
• Score: 24.30721158142313
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fragmented safety evaluation undermines the governance of dangerous AI capabilities. We present a modular framework that evaluates each model through three orthogonal pipelines---Knowledge ($K$), Defense ($D$), and Harm ($H$)---under a unified protocol, aggregating results into a standardized dangerous-capability profile $φ$. Pluggable modules supply scenario seeds, knowledge banks, hazard queries, and judge rubrics, while the core evaluation engine remains unchanged across domains; the CB evaluation is complemented by a cyber pilot demonstrating protocol transfer. Instantiating the framework with a chemical-biological (CB) module, we evaluate 12 commercial LLMs from four families. Our first contribution is a horizontal comparison of dangerous capability across models and model families: the three dimensions expose sharply divergent profiles---models with comparable knowledge differ in refusal resilience, and strong defenders do not generate less harmful content when they do comply---while family-level patterns further separate Claude, DeepSeek, and GPT models. The second is a temporal analysis of capability evolution: tracking $K$, $D$, and $H$ against model release dates reveals that dangerous capability has not monotonically declined; newer models deepen knowledge while only partially improving defense, showing that scaling and alignment progress do not uniformly translate into safety. Reliability is established via cross-judge consistency (bootstrap $ρ> 0.79$, 4 of 5 judges) and pipeline orthogonality ($K$--$D$--$H$ inter-correlations $ρ\in [0.32, 0.52]$).

Related papers

• AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval [1.1647715835517187]
  AgentWorld is a simulation framework combining personality-driven user populations with stateful tool-use environments.<n>Three experiments demonstrate the framework: a conversational analytics agent across 10 OCEAN personas (240 evaluator judgments); a customer-support agent across 5 tasks $times$ 4 persona variants; and adversarial stress-testing of 5 tasks revealing pre-existing trajectory brittleness.
  arXiv  Detail & Related papers  (2026-08-25T05:15:33Z)
• CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs [100.38986535324284]
  We introduce textbfCIAware-Bench, a benchmark for measuring textbfcontrol textbfintervention (CI) awareness across frontier models.<n>CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention.
  arXiv  Detail & Related papers  (2026-06-09T16:24:16Z)
• Persona-Model Collapse in Emergent Misalignment [0.0]
  Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts.<n>We test this hypothesis using two metrics: moral susceptibility (S) and moral robustness (R)<n>These metrics formalize the model's ability to differentiate characters (S) and its consistency when simulating a given one (R)
  arXiv  Detail & Related papers  (2026-05-13T00:48:57Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Empowering VLMs for Few-Shot Multimodal Time Series Classification via Tailored Agentic Reasoning [44.69766273765273]
  We propose the first VL$underlinetextbfM$ $underlinetextbfa$gentic $underlinetextbfr$easoning framework for few-$underlinetextbfs$hot multimodal $underlinetextbfT$ime $underlinetextbfS$eries $underlinetextbfC$lassification ($textbfMarsTSC$), which introduces a self-evolving knowledge bank as a dynamic context iteratively refined via reflective agent
  arXiv  Detail & Related papers  (2026-05-10T07:47:09Z)
• Agentic Adversarial Rewriting Exposes Architectural Vulnerabilities in Black-Box NLP Pipelines [34.37967130277969]
  We propose a two-agent evasion framework operating in a semantic perturbation space.<n>Evasion rates are 19.95 to 40.34% on modern large language model (LLM) based systems.<n>A pattern-informed defense reduces the evasion rate by up to 65.18%.
  arXiv  Detail & Related papers  (2026-04-26T00:58:36Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis [96.92417622318267]
  ATBench is a trajectory-level benchmark for structured, diverse, and realistic evaluation of agent safety.<n>It organizes agentic risk along three dimensions: risk source, failure mode, and real-world harm.<n>It contains 1,000 trajectories (503 safe and 497 unsafe), averaging 9.01 turns and 3.95k tokens, with 1,954 invoked tools drawn from pools spanning 2,084 available tools.
  arXiv  Detail & Related papers  (2026-04-02T13:26:20Z)
• AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective [12.53973720107764]
  We propose a closed-loop threat taxonomy that frames model-data interactions along four axes.<n>The resulting four classes of security threats represent distinct but interrelated categories of attacks.<n>Our framework establishes a foundation for developing scalable, transferable, and cross-modal security strategies.
  arXiv  Detail & Related papers  (2026-03-25T22:53:43Z)
• PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach [49.14349403242654]
  We present $textbfPropensityBench$, a novel benchmark framework that assesses the proclivity of models to engage in risky behaviors.<n>Our framework includes 5,874 scenarios with 6,648 tools spanning four high-risk domains: cybersecurity, self-proliferation, biosecurity, and chemical security.<n>Across open-source and proprietary frontier models, we uncover 9 alarming signs of propensity: models frequently choose high-risk tools when under pressure.
  arXiv  Detail & Related papers  (2025-11-24T18:46:44Z)
• SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models [66.71948519280669]
  Multimodal Large Reasoning Models (MLRMs) demonstrate impressive crossmodal reasoning but often amplify safety risks under adversarial prompts.<n> Existing defenses mainly act at the output level and do not constrain the reasoning process, leaving models to implicit risks.<n>We propose SaFeR-VLM, which integrates four components and supports dynamic and interpretable safety decisions beyond surface-level filtering.
  arXiv  Detail & Related papers  (2025-10-08T10:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.