Trivial Prompt Reframing Bypasses Safety Guardrails in Googleś MedGemma-4B

• URL: http://arxiv.org/abs/2607.09804v1
• Date: Thu, 09 Jul 2026 18:31:43 GMT
• Title: Trivial Prompt Reframing Bypasses Safety Guardrails in Googleś MedGemma-4B
• Abstract summary: Open-weight medical language models are increasingly used as the base of patient-facing and clinician-support applications.<n>We quantify that gap for MedGemma-4B-it under attacks that require no technical sophistication.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-weight medical language models are increasingly used as the base of patient-facing and clinician-support applications. Their model cards prohibit specific behaviors -- recommending exact drug dosages, issuing definitive diagnoses, prescribing treatments, adjudicating drug-drug interactions, and advising that emergency care can be skipped -- yet a model card describes intended behavior, not robust behavior. We quantify that gap for MedGemma-4B-it under attacks that require no technical sophistication. We build a fully factorial benchmark of 5 guarded-behavior concepts x 50 deterministically templated questions x 6 lay-accessible attack manners x 3 repetitions (4,500 generations), serve the model locally through Ollama under default sampling, and code every response refuse/hedge/comply with three independent judges (an LLM judge, a transparent regex judge, and an NLI-entailment judge). Under the primary LLM judge the overall Attack Success Rate (ASR, the fraction coded comply) is 38.0%. The two framings that reinterpret the request as legitimate dominate: recasting a question as a "medical board exam" item raises ASR from a 29.0% baseline to 53.1% (+24.0 points), and an appeal to an alleged doctor's authority raises it to 43.7% (+14.7); crude instruction-override prefixes have no significant effect. Robustness is dominated by topic: the drug-interaction guardrail is nearly absent (83.2% ASR) while the emergency-deferral guardrail is strong (4.7%) -- and the authority framing is the only attack that breaches it. We report Wilson confidence intervals, cluster-bootstrap effect sizes, a cluster-robust logistic regression, Cochran's Q, per-manner McNemar tests, and inter-judge reliability (Fleiss' kappa = 0.26); absolute ASR is judge-dependent while the ordering of attacks and topics is not. Our findings motivate stronger deployment-time guardrails for open medical models.

Related papers

• Beyond the Flag: Clinical Framing Closes the Moderation Gap in Suicide Risk Measurement [0.0]
  We ask how well deployed safety signals recover clinically meaningful severity.<n>We release a benchmark of 516 r/Watch posts rated by a licensed psychiatrist on a four-level ordinal schema.<n>We argue graded severity, not a binary flag, is what a proportionate duty of care requires.
  arXiv  Detail & Related papers  (2026-09-05T21:16:16Z)
• Cybersecurity Detection Classification with Reasoning-enabled Language Models [50.72675435043546]
  A major issue in Security Operations Centers (SOCs) is alert fatigue.<n>Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat.<n>We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections.
  arXiv  Detail & Related papers  (2026-07-30T16:22:13Z)
• Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety [0.0]
  We stress-test four models by deleting half the final user turn in HealthBench conversations, grading responses with a four-provider LLM-judge panel and a blinded clinician-anchored reference.<n>Two evaluator-facing results are robust. First, judge choice materially changes apparent safety.<n>Second, LLM judges are more permissive than clinicians on a blinded 50-item subsample.
  arXiv  Detail & Related papers  (2026-07-21T08:05:35Z)
• PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models [20.805546078350524]
  PsychoSafe reframes refusal as structured supportive communication grounded in evidence-based intervention strategies.<n>On a validation set of 500 prompts, PsychoSafe prompting improves overall refusal quality by 28.1% over a generic baseline.<n>Fine-tuning achieves near-perfect refusal and resource-referral rates but reduces response relevance.
  arXiv  Detail & Related papers  (2026-06-08T16:19:18Z)
• The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems [1.0262304700896199]
  We formalize emphSemantic Norm Drift (SND) as a third path to agent misconduct.<n>In SND, a policy-formatted document enters a shared vector store through normal uploads and later reappears as trusted system context.<n>Counterfactual Composition Testing identifies the causal entry with 87.5% accuracy and zero false positives.
  arXiv  Detail & Related papers  (2026-05-12T20:21:47Z)
• ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV [0.0]
  Reasoning benchmarks measure clinical performance on clean inputs.<n>We evaluate the step before reasoning: retrieval over real EHR notes, where negation, temporality, and family-versus-patient attribution can flip a correct answer to a wrong one.<n>EpiKG carries assertion label and a temporality tag with every fact in a patient knowledge graph, then routes retrieval by question intent.
  arXiv  Detail & Related papers  (2026-05-11T18:47:52Z)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents [2.1942030377331245]
  Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets.<n>We introduce MOSAIC-Bench, a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates.<n>We show that nine production coding agents compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs.
  arXiv  Detail & Related papers  (2026-05-05T16:38:23Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context [82.32380418146656]
  Health-ORSC-Bench is the first large-scale benchmark designed to measure textbfOver-Refusal and textbfSafe Completion quality in healthcare.<n>Our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity.<n>Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants.
  arXiv  Detail & Related papers  (2026-01-25T01:28:52Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• CausalDiff: Causality-Inspired Disentanglement via Diffusion Model for Adversarial Defense [61.78357530675446]
  Humans are difficult to be cheated by subtle manipulations, since we make judgments only based on essential factors.<n>Inspired by this observation, we attempt to model label generation with essential label-causative factors and incorporate label-non-causative factors to assist data generation.<n>For an adversarial example, we aim to discriminate perturbations as non-causative factors and make predictions only based on the label-causative factors.
  arXiv  Detail & Related papers  (2024-10-30T15:06:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.