More Agents Helps but Adversarial Robustness Gap Persists

• URL: http://arxiv.org/abs/2511.07112v1
• Date: Mon, 10 Nov 2025 13:58:17 GMT
• Title: More Agents Helps but Adversarial Robustness Gap Persists
• Abstract summary: When LLM agents work together, they seem to be more powerful than a single LLM in mathematical question answering.<n>We investigate this question using adversarially perturbed math questions.
• Score: 10.507110902601466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When LLM agents work together, they seem to be more powerful than a single LLM in mathematical question answering. However, are they also more robust to adversarial inputs? We investigate this question using adversarially perturbed math questions. These perturbations include punctuation noise with three intensities (10, 30, and 50 percent), plus real-world and human-like typos (WikiTypo, R2ATA). Using a unified sampling-and-voting framework (Agent Forest), we evaluate six open-source models (Qwen3-4B/14B, Llama3.1-8B, Mistral-7B, Gemma3-4B/12B) across four benchmarks (GSM8K, MATH, MMLU-Math, MultiArith), with various numbers of agents n from one to 25 (1, 2, 5, 10, 15, 20, 25). Our findings show that (1) Noise type matters: punctuation noise harm scales with its severity, and the human typos remain the dominant bottleneck, yielding the largest gaps to Clean accuracy and the highest ASR even with a large number of agents. And (2) Collaboration reliably improves accuracy as the number of agents, n, increases, with the largest gains from one to five agents and diminishing returns beyond 10 agents. However, the adversarial robustness gap persists regardless of the agent count.

Related papers

• AgentFairBench: Do LLM Agents Discriminate When They Act? [2.3004655342211078]
  AgentFairBench is a cheap, reproducible, multi-domain benchmark for demographic disparity in the actions of LLM agents.<n>It spans three regulator-anchored domains: hiring, lending, and medical triage.<n>A NumPy-only harness computes counterfactual flip rate, mean absolute score difference (MASD), action-rate disparity, and tool-invocation disparity.
  arXiv  Detail & Related papers  (2026-06-15T13:50:26Z)
• Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps [0.0]
  Frontier deep research agents (DRAs) plan a research task, synthesize across documents, and return a structured deliverable on demand.<n>Existing benchmarks measure factual recall, single-hop QA, or generic agentic skill, missing the multi-document, decision-grade work DRAs are deployed to produce.<n>We grade three frontier agents, namely Claude Opus 4.6 with web search, OpenAI o3-deep-research, and Google Gemini 3.1 Pro deep-research, on 42 SME-authored prompts.
  arXiv  Detail & Related papers  (2026-05-17T17:32:52Z)
• MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments [68.15682316179088]
  We introduce MERRIN (Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments), a human-annotated benchmark for evaluating search-augmented agents.<n> MERRIN measures AI agents' ability to identify relevant modalities, retrieve multimodal evidence, and perform multi-hop reasoning over noisy web sources.<n>We evaluate diverse search agents powered by ten models, including strong closed-source models and open-weight models.
  arXiv  Detail & Related papers  (2026-04-15T02:37:47Z)
• AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent [57.10083973844841]
  AgentArk is a novel framework to distill multi-agent dynamics into the weights of a single model.<n>We investigate three hierarchical distillation strategies across various models, tasks, scaling, and scenarios.<n>By shifting the burden of computation from inference to training, the distilled models preserve the efficiency of one agent while exhibiting strong reasoning and self-correction performance of multiple agents.
  arXiv  Detail & Related papers  (2026-02-03T19:18:28Z)
• Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests [0.0]
  We analyze 33,596 agent-generated PRs and 6,618 human PRs to compare code-change characteristics and message quality.<n>Agents generate stronger commit-level messages but lag humans at PR-level summarization.<n>These findings highlight a gap between agents' micro-level precision and macro-level communication.
  arXiv  Detail & Related papers  (2026-01-24T23:33:07Z)
• Multi-Agent Code Verification with Compound Vulnerability Detection [0.0]
  Existing tools only catch 65% of bugs with 35% false positives.<n>We built CodeX-Verify, a multi-agent system that uses four specialized agents to detect different types of bugs.
  arXiv  Detail & Related papers  (2025-11-20T03:40:27Z)
• Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks [10.7231991032233]
  Large Language Model (LLM) agents are powering a growing share of interactive web applications, yet remain vulnerable to misuse and harm.<n>We present the Online Harassment Agentic Benchmark consisting of: (i) a synthetic multi-turn harassment conversation dataset, (ii) a multi-agent (e.g., harasser, victim) simulation informed by repeated game theory, (iii) three jailbreak methods attacking agents across memory, planning, and fine-tuning, and (iv) a mixed-methods evaluation framework.
  arXiv  Detail & Related papers  (2025-10-16T01:27:44Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoning [11.045096250408067]
  Tree of Agents (TOA) is a multi-agent reasoning framework that segments the input into chunks processed by independent agents.<n>TOA enables agents to probe different reasoning orders for multi-perspective understanding.<n>To improve processing efficiency, we incorporate prefix-hash caching and adaptive pruning strategies.
  arXiv  Detail & Related papers  (2025-09-08T08:34:02Z)
• Know the Ropes: A Heuristic Strategy for LLM-based Multi-Agent System Design [27.246783209927464]
  Know-The-Ropes is a framework that converts domain priors into an algorithmic blueprint hierarchy.<n>Three GPT-4o-mini agents raise accuracy from 3% zero-shot to 95% on size-5 instances after patching a single bottleneck agent.
  arXiv  Detail & Related papers  (2025-05-22T17:52:33Z)
• Retrieval-Augmented Generation with Conflicting Evidence [57.66282463340297]
  Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses.<n>In practice, these systems often need to handle ambiguous user queries and potentially conflicting information from multiple sources.<n>We propose RAMDocs (Retrieval with Ambiguity and Misinformation in Documents), a new dataset that simulates complex and realistic scenarios for conflicting evidence for a user query.
  arXiv  Detail & Related papers  (2025-04-17T16:46:11Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>The impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents on different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• Malicious Agent Detection for Robust Multi-Agent Collaborative Perception [52.261231738242266]
  Multi-agent collaborative (MAC) perception is more vulnerable to adversarial attacks than single-agent perception. We propose Malicious Agent Detection (MADE), a reactive defense specific to MAC perception. We conduct comprehensive evaluations on a benchmark 3D dataset V2X-sim and a real-road dataset DAIR-V2X.
  arXiv  Detail & Related papers  (2023-10-18T11:36:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.