AgentSim: A Platform for Verifiable Agent-Trace Simulation

• URL: http://arxiv.org/abs/2604.26653v1
• Date: Wed, 29 Apr 2026 13:19:38 GMT
• Title: AgentSim: A Platform for Verifiable Agent-Trace Simulation
• Abstract summary: AgentSim is an open-source platform for simulating RAG agents.<n>It generates verifiable, stepwise traces of agent reasoning over any document collection.<n>It combines a multi-model validation pipeline with an active human-in-the-loop process.
• Score: 3.2058241360543254
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training trustworthy agentic LLMs requires data that shows the grounded reasoning process, not just the final answer. Existing datasets fall short: question-answering data is outcome-only, chain-of-thought data is not tied to specific documents, and web-agent datasets track interface actions rather than the core retrieval and synthesis steps of a RAG workflow. We introduce AgentSim, an open-source platform for simulating RAG agents. It generates verifiable, stepwise traces of agent reasoning over any document collection. AgentSim uses a policy to ensure the agent widely explores the document set. It combines a multi-model validation pipeline with an active human-in-the-loop process. This approach focuses human effort on difficult steps where models disagree. Using AgentSim, we construct and release the Agent-Trace Corpus (ATC), a large collection of grounded reasoning trajectories spanning three established IR benchmarks. We make three contributions: (1) the AgentSim platform with two mechanisms, Corpus-Aware Seeding and Active Validation, that improve trace diversity and quality; (2) the Agent-Trace Corpus (ATC), over 103,000 verifiable reasoning steps spanning three IR benchmarks, with 100% grounding rate on substantive answers; and (3) a comparative behavioral analysis revealing systematic differences in how state-of-the-art models approach information seeking. Platform, toolkit, and corpus are publicly available.

Related papers

• Agentic AI and Retrieval-Augmented Models in Straight-Through Underwriting [7.670757733138505]
  Actuaries now face a design space that ranges from traditional rule-based automation to large language models.<n>This paper examines how these emerging architectures can support actuarial priorities such as transparency, auditability, and human-in-the-loop governance.
  arXiv  Detail & Related papers  (2026-07-08T18:43:34Z)
• PRIMA: Operational Patterns for Resilient Multi-Agent Research with Verifiable Identity and Convergent Feedback [0.0]
  PRIMA operates as coordinated multi-agent research systems over multi-hour runs.<n>Main contributions are three operational patterns for surviving failure modes.<n>A Graph Isomorphism case study grounds the architectural claims in a generated artifact.
  arXiv  Detail & Related papers  (2026-05-23T23:27:46Z)
• Towards Multi-Agent Autonomous Reasoning in Hydrodynamics [0.06999740786886537]
  We present a multi-agent system (MAS) prototype for hydrodynamics in which specialized agents are coordinated through a Layer Execution Graph (LEG)<n>A planner agent constructs query-specific execution topologies from natural-language routings that capture domain knowledge without hard-coding it as rigid control logic.<n>A reporter agent synthesizes the final response, while the runtime logs provenance for every tool invocation to support auditability.
  arXiv  Detail & Related papers  (2026-05-01T21:17:55Z)
• AgentIR: Reasoning-Aware Retrieval for Deep Research Agents [76.29382561831105]
  Deep Research agents generate explicit natural language reasoning before each search call.<n> Reasoning-Aware Retrieval embeds the agent's reasoning trace alongside its query.<n>DR- Synth generates Deep Research retriever training data from standard QA datasets.<n>AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch.
  arXiv  Detail & Related papers  (2026-03-04T18:47:26Z)
• AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation [39.61543921719145]
  AgentSelect is a benchmark that reframes agent selection as narrative query-to-agent recommendation.<n>It converts heterogeneous evaluation artifacts into unified, positive-only interaction data.<n>AgentSelect provides the first unified data and evaluation infrastructure for agent recommendation.
  arXiv  Detail & Related papers  (2026-03-04T06:17:51Z)
• Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback [51.22403664895878]
  Agent2World is a tool-augmented multi-agent framework that achieves strong inference-time world-model generation.<n>It also serves as a data engine for supervised fine-tuning, by grounding generation in multi-agent feedback.
  arXiv  Detail & Related papers  (2025-12-26T18:54:14Z)
• MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles [25.198556596878362]
  We introduce three key strategies for parameter-efficient fine-tuning (PEFT) in agent tasks.<n>Inspired by the increasingly dominant Reason+Action paradigm, we first decompose the capabilities necessary for the agent tasks into three distinct roles.<n>We propose the Mixture-of-Roles (MoR) framework, which comprises three specialized Low-Rank Adaptation (LoRA) groups, each designated to fulfill a distinct role.
  arXiv  Detail & Related papers  (2025-12-25T15:02:07Z)
• Multi-Agent Systems for Dataset Adaptation in Software Engineering: Capabilities, Limitations, and Future Directions [8.97512410819274]
  This paper presents the first empirical study on how state-of-the-art multi-agent systems perform in dataset adaptation tasks.<n>We evaluate GitHub Copilot on adapting SE research artifacts from benchmark repositories including ROCODE and LogHub2.0.<n>Results show that current systems can identify key files and generate partial adaptations but rarely produce correct implementations.
  arXiv  Detail & Related papers  (2025-11-26T13:26:11Z)
• MCP-Orchestrated Multi-Agent System for Automated Disinformation Detection [84.75972919995398]
  This paper presents a multi-agent system that uses relation extraction to detect disinformation in news articles.<n>The proposed Agentic AI system combines four agents: (i) a machine learning agent (logistic regression), (ii) a Wikipedia knowledge check agent, and (iv) a web-scraped data analyzer.<n>Results demonstrate that the multi-agent ensemble achieves 95.3% accuracy with an F1 score of 0.964, significantly outperforming individual agents and traditional approaches.
  arXiv  Detail & Related papers  (2025-08-13T19:14:48Z)
• AgentRE: An Agent-Based Framework for Navigating Complex Information Landscapes in Relation Extraction [10.65417796726349]
  relation extraction (RE) in complex scenarios faces challenges such as diverse relation types and ambiguous relations between entities within a single sentence. We propose an agent-based RE framework, namely AgentRE, which fully leverages the potential of large language models to achieve RE in complex scenarios.
  arXiv  Detail & Related papers  (2024-09-03T12:53:05Z)
• Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models [56.00992369295851]
  Open-sourced Large Language Models (LLMs) have achieved great success in various NLP tasks, however, they are still far inferior to API-based models when acting as agents. This paper delivers three key observations: (1) the current agent training corpus is entangled with both formats following and agent reasoning, which significantly shifts from the distribution of its pre-training data; (2) LLMs exhibit different learning speeds on the capabilities required by agent tasks; and (3) current approaches have side-effects when improving agent abilities by introducing hallucinations. We propose Agent-FLAN to effectively Fine-tune LANguage models for Agents.
  arXiv  Detail & Related papers  (2024-03-19T16:26:10Z)
• Zero-shot Composed Text-Image Retrieval [72.43790281036584]
  We consider the problem of composed image retrieval (CIR) It aims to train a model that can fuse multi-modal information, e.g., text and images, to accurately retrieve images that match the query, extending the user's expression ability.
  arXiv  Detail & Related papers  (2023-06-12T17:56:01Z)
• Retrieval-Augmented Reinforcement Learning [63.32076191982944]
  We train a network to map a dataset of past experiences to optimal behavior. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context. We show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores.
  arXiv  Detail & Related papers  (2022-02-17T02:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.