Misalignment Bounty: Crowdsourcing AI Agent Misbehavior

• URL: http://arxiv.org/abs/2510.19738v2
• Date: Wed, 05 Nov 2025 13:34:49 GMT
• Title: Misalignment Bounty: Crowdsourcing AI Agent Misbehavior
• Abstract summary: Misalignment Bounty collected cases of agents pursuing unintended or unsafe goals.<n>The bounty received 295 submissions, of which nine were awarded.<n>This report explains the program's motivation and evaluation criteria.
• Score: 0.16100415819628258
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advanced AI systems sometimes act in ways that differ from human intent. To gather clear, reproducible examples, we ran the Misalignment Bounty: a crowdsourced project that collected cases of agents pursuing unintended or unsafe goals. The bounty received 295 submissions, of which nine were awarded. This report explains the program's motivation and evaluation criteria, and walks through the nine winning submissions step by step.

Related papers

• Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents [0.0]
  Agentic red-teaming benchmarks report whether an injected agent was compromised as a single bit: the attack succeeded, or it did not.<n>We argue that this binary attack-success rate discards the information a defender most needs, namely how harmful the resulting action was.<n>We introduce an action-graded harm rubric that scores an agent's tool-call trajectory on a seven-level ordinal scale (L0 to L6) according to whether the executed action was reversible, whether it crossed scope to reach another party, and whether it expanded privilege.
  arXiv  Detail & Related papers  (2026-07-08T14:38:01Z)
• The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity? [53.15349353876531]
  As AI becomes more capable, we entrust it with more general and consequential tasks.<n>We operationalize this question using a bias-variance decomposition of the errors made by AI models.<n>As more capable AIs pursue harder tasks, requiring more sequential action and thought, our results predict failures to be accompanied by more incoherent behavior.
  arXiv  Detail & Related papers  (2026-01-30T14:52:03Z)
• The Current State of AI Bias Bounties: An Overview of Existing Programmes and Research [18.854429764918734]
  Current bias evaluation methods rarely engage with communities impacted by AI systems.<n>Inspired by bug bounties, bias bounties have been proposed as a reward-based method.
  arXiv  Detail & Related papers  (2025-10-02T14:09:11Z)
• Deceptive Sequential Decision-Making via Regularized Policy Optimization [54.38738815697299]
  We model autonomous systems as Markov decision processes, with adversaries using inverse reinforcement learning to recover reward functions.<n>We show three regularization strategies for policy synthesis problems that actively deceive an adversary about a system's reward.<n>We show that diversionary, targeted, and equivocal deception all steer the adversary to false beliefs while still attaining a total accumulated reward that is at least 97% of its optimal, non-deceptive value.
  arXiv  Detail & Related papers  (2025-01-30T23:41:40Z)
• Jailbreaking as a Reward Misspecification Problem [80.52431374743998]
  We propose a novel perspective that attributes this vulnerability to reward misspecification during the alignment process.<n>We introduce a metric ReGap to quantify the extent of reward misspecification and demonstrate its effectiveness.<n>We present ReMiss, a system for automated red teaming that generates adversarial prompts in a reward-misspecified space.
  arXiv  Detail & Related papers  (2024-06-20T15:12:27Z)
• Poisoning Retrieval Corpora by Injecting Adversarial Passages [79.14287273842878]
  We propose a novel attack for dense retrieval systems in which a malicious user generates a small number of adversarial passages. When these adversarial passages are inserted into a large retrieval corpus, we show that this attack is highly effective in fooling these systems. We also benchmark and compare a range of state-of-the-art dense retrievers, both unsupervised and supervised.
  arXiv  Detail & Related papers  (2023-10-29T21:13:31Z)
• Bound by the Bounty: Collaboratively Shaping Evaluation Processes for Queer AI Harms [14.872177285434999]
  We ask queer communities for their positions on, and desires from, auditing processes. We organized a participatory workshop to critique and bias bounties from queer perspectives. We conclude by advocating for community ownership of bounties and complementing bounties with participatory processes.
  arXiv  Detail & Related papers  (2023-07-15T00:49:45Z)
• Experiments with Detecting and Mitigating AI Deception [0.0]
  How to detect and deceptive AI systems is an open problem for the field of safe and trustworthy AI. We analyse two algorithms for mitigating deception: The first is based on the path-specific objectives framework where paths in the game that incentivise deception are removed. The second is based on shielding, i.e., monitoring for unsafe policies and replacing them with a safe reference policy.
  arXiv  Detail & Related papers  (2023-06-26T16:22:13Z)
• SHIELD: Thwarting Code Authorship Attribution [11.311401613087742]
  Authorship attribution has become increasingly accurate, posing a serious privacy risk for programmers who wish to remain anonymous. We introduce SHIELD to examine the robustness of different code authorship attribution approaches against adversarial code examples.
  arXiv  Detail & Related papers  (2023-04-26T02:55:28Z)
• Formalizing the Problem of Side Effect Regularization [81.97441214404247]
  We propose a formal criterion for side effect regularization via the assistance game framework. In these games, the agent solves a partially observable Markov decision process. We show that this POMDP is solved by trading off the proxy reward with the agent's ability to achieve a range of future tasks.
  arXiv  Detail & Related papers  (2022-06-23T16:36:13Z)
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models [85.68751244243823]
  Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. We find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward.
  arXiv  Detail & Related papers  (2022-01-10T18:58:52Z)
• MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning [65.52675802289775]
  We show that an uncertainty aware classifier can solve challenging reinforcement learning problems. We propose a novel method for computing the normalized maximum likelihood (NML) distribution. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions.
  arXiv  Detail & Related papers  (2021-07-15T08:19:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.