AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval

• URL: http://arxiv.org/abs/2608.24076v2
• Date: Wed, 26 Aug 2026 06:11:33 GMT
• Title: AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval
• Abstract summary: AgentWorld is a simulation framework combining personality-driven user populations with stateful tool-use environments.<n>Three experiments demonstrate the framework: a conversational analytics agent across 10 OCEAN personas (240 evaluator judgments); a customer-support agent across 5 tasks $times$ 4 persona variants; and adversarial stress-testing of 5 tasks revealing pre-existing trajectory brittleness.
• Score: 1.1647715835517187
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluation of agentic information retrieval remains limited to scripted interactions with uniform users, missing both natural personality diversity and adversarial brittleness. We present AgentWorld, a simulation framework combining (i)Big Five (OCEAN) personality-driven user populations with stateful tool-use environments; (ii)the pass$^k$ consistency metric with structured fault classification, partial-credit scoring, and dual-control handoff verification; (iii)score-thresholded training-data export in six fine-tuning formats; and (iv)an adversarial Risk Analyser that snapshots required-intermediate-state spines, branches Monte-Carlo rollouts under four task-aware perturbation types, and quantifies risk via $ΔP / ΔT$ scoring, Dempster--Shafer evidence fusion, and Shapley attack-category attribution. Three experiments demonstrate the framework: a conversational analytics agent across 10 OCEAN personas (240 evaluator judgments); a customer-support agent across 5 tasks $\times$ 4 persona variants; and adversarial stress-testing of 5 tasks revealing pre-existing trajectory brittleness ($V_{\min}=0.375$ without perturbation) and tool/infrastructure-layer attack dominance (Shapley: 46% system, 38% action). Personality variation surfaces failure modes uniform testing cannot expose---cross-domain leakage, contextual drift, a 0.27-point quality gap, and 50% vs. 100% pass-rate across personas on the same task---while the Risk Analyser quantifies trajectory-level brittleness that pass$^k$ alone cannot measure.

Related papers

• FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs [24.30721158142313]
  Fragmented safety evaluation undermines the governance of dangerous AI capabilities.<n>We present a modular framework that evaluates each model through three orthogonal pipelines---Knowledge, Defense, and Harm.<n>Our first contribution is a horizontal comparison of dangerous capability across models and model families.<n>The second is a temporal analysis of capability evolution: tracking $K$, $D$, and $H$ against model release dates reveals that dangerous capability has not monotonically declined.
  arXiv  Detail & Related papers  (2026-09-02T06:31:39Z)
• Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations [0.0]
  We show, across three open agent-trace benchmarks, that the agent main effect accounts for less than 3% of total variance in every dataset and check type.<n>We arrive at this through a four-facet Generalizability Theory variance decomposition, fit with three estimators.<n>We package these into Deployment Decision Reliability (DDR), a one-page reporting discipline that turns the variance-component table into five decisions an enterprise buyer can defend.
  arXiv  Detail & Related papers  (2026-08-11T18:16:51Z)
• BRiG-AFA: Bellman Risk-to-Go Learning for Non-Myopic Active Feature Acquisition [8.363536351727978]
  Active feature acquisition asks which unobserved feature to measure next for each test instance under a budget.<n>We introduce method, a deployable, supervised alternative that learns a separate candidate-conditioned risk-to-go function for every remaining budget.
  arXiv  Detail & Related papers  (2026-08-03T14:30:44Z)
• Persona-Model Collapse in Emergent Misalignment [0.0]
  Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts.<n>We test this hypothesis using two metrics: moral susceptibility (S) and moral robustness (R)<n>These metrics formalize the model's ability to differentiate characters (S) and its consistency when simulating a given one (R)
  arXiv  Detail & Related papers  (2026-05-13T00:48:57Z)
• Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems [0.0]
  Agent skills expose both executable behavior and context-setting documentation.<n>A realistic attacker can use audit and runtime feedback to repeatedly rewrite the skill.<n>Proteus searches a formalized five-axis skill-attack space.
  arXiv  Detail & Related papers  (2026-05-12T10:05:54Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators [0.0]
  We introduce AgentCollabBench, a diagnostic benchmark of 900 human-validated tasks spanning software engineering, DevOps, and data engineering.<n>Each task isolates one of four behavioral risks: instruction decay, false-belief contagion, context leakage, and tracer durability.<n>We evaluate four modern LLMs (GPT 4.1 mini, Gemini 2.5 Flash Lite, Qwen-3.5-35B-A3B, and Llama 3.1 8B Instruct)<n> communication topology emerges as a primary risk factor that explains 7-40% of the variance in multi-hop information survival.
  arXiv  Detail & Related papers  (2026-05-09T03:35:09Z)
• TRUST: A Framework for Decentralized AI Service v.0.1 [47.384270414446604]
  Large Reasoning Models (LRMs) and Multi-Agent Systems (MAS) in high-stakes domains demand reliable verification.<n>We introduce TRUST (Transparent, Robust, and Unified Services for Trustworthy AI), a decentralized framework with three innovations.<n>We prove a Safety-Profitability Theorem ensuring honest auditors profit while malicious actors incur losses.
  arXiv  Detail & Related papers  (2026-04-29T19:32:58Z)
• OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation [57.505743202759646]
  OccuBench is a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains.<n>Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity.
  arXiv  Detail & Related papers  (2026-04-13T00:27:32Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>The impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents on different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.