Get Experience from Practice: LLM Agents with Record & Replay

• URL: http://arxiv.org/abs/2505.17716v1
• Date: Fri, 23 May 2025 10:33:14 GMT
• Title: Get Experience from Practice: LLM Agents with Record & Replay
• Abstract summary: This paper proposes a new paradigm called AgentRR (Agent Record & Replay), which introduces the classical record-and-replay mechanism into AI agent frameworks.<n>We detail a multi-level experience abstraction method and a check function mechanism in AgentRR.<n>In addition, we explore multiple application modes of AgentRR, including user-recorded task demonstration, large-small model collaboration and privacy-aware agent execution.
• Score: 16.179801770737892
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI agents, empowered by Large Language Models (LLMs) and communication protocols such as MCP and A2A, have rapidly evolved from simple chatbots to autonomous entities capable of executing complex, multi-step tasks, demonstrating great potential. However, the LLMs' inherent uncertainty and heavy computational resource requirements pose four significant challenges to the development of safe and efficient agents: reliability, privacy, cost and performance. Existing approaches, like model alignment, workflow constraints and on-device model deployment, can partially alleviate some issues but often with limitations, failing to fundamentally resolve these challenges. This paper proposes a new paradigm called AgentRR (Agent Record & Replay), which introduces the classical record-and-replay mechanism into AI agent frameworks. The core idea is to: 1. Record an agent's interaction trace with its environment and internal decision process during task execution, 2. Summarize this trace into a structured "experience" encapsulating the workflow and constraints, and 3. Replay these experiences in subsequent similar tasks to guide the agent's behavior. We detail a multi-level experience abstraction method and a check function mechanism in AgentRR: the former balances experience specificity and generality, while the latter serves as a trust anchor to ensure completeness and safety during replay. In addition, we explore multiple application modes of AgentRR, including user-recorded task demonstration, large-small model collaboration and privacy-aware agent execution, and envision an experience repository for sharing and reusing knowledge to further reduce deployment cost.

Related papers

• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
  The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
  arXiv  Detail & Related papers  (2026-01-28T13:49:18Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systems [0.0]
  Recent advances in agentic AI have shifted the focus from standalone Large Language Models to integrated systems.<n>We propose an end-to-end Agent Assessment Framework with four evaluation pillars encompassing LLMs, Memory, Tools, and Environment.<n>We validate the framework on a representative Autonomous CloudOps use case, where experiments reveal behavioral deviations by conventional metrics.
  arXiv  Detail & Related papers  (2025-12-14T18:17:40Z)
• Formalizing the Safety, Security, and Functional Properties of Agentic AI Systems [10.734711935895225]
  We introduce a modeling framework for agentic AI systems composed of two foundational models.<n>The first, the host agent model, formalizes the top-level entity that interacts with the user, decomposes tasks, and orchestrates their execution by leveraging external agents and tools.<n>The second, the task lifecycle model, details the states and transitions of individual sub-tasks from creation to completion, providing a fine-grained view of task management and error handling.
  arXiv  Detail & Related papers  (2025-10-15T22:02:30Z)
• RA-Gen: A Controllable Code Generation Framework Using ReAct for Multi-Agent Task Execution [18.209965705151905]
  We propose a controllable code generation framework utilizing the ReAct paradigm for multi-agent task execution.<n>This framework is a multi-agent system designed to enable efficient, precise, and interpretable code generation.
  arXiv  Detail & Related papers  (2025-10-09T15:59:24Z)
• InfiAgent: Self-Evolving Pyramid Agent Framework for Infinite Scenarios [28.65914611521654]
  InfiAgent is a Pyramid-like DAG-based Multi-Agent Framework that can be applied to textbfinfinite scenarios.<n>InfiAgent achieves 9.9% higher performance compared to ADAS (similar auto-generated agent framework)
  arXiv  Detail & Related papers  (2025-09-26T15:44:09Z)
• AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production [4.031479494871582]
  We present Agent, the first evaluation framework designed specifically for post-deployment monitoring and reasoning of agentic pipeline.<n>Agent achieves state-of-the-art results on key metrics, while uncovering critical issues missed in human annotations.
  arXiv  Detail & Related papers  (2025-09-18T05:59:04Z)
• GoalfyMax: A Protocol-Driven Multi-Agent System for Intelligent Experience Entities [4.406205045227101]
  We present GoalfyMax, a protocol-driven framework for end-to-end multi-agent collaboration.<n>GoalfyMax introduces a standardized Agent-to-Agent (A2A) communication layer built on the Model Context Protocol (MCP)<n>It incorporates the Experience Pack (XP) architecture, a layered memory system that preserves both task rationales and execution traces.
  arXiv  Detail & Related papers  (2025-07-13T05:13:52Z)
• SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agents [58.21223208538351]
  This work explores the security issues surrounding mobile multimodal agents.<n>It attempts to construct a risk discrimination mechanism by incorporating behavioral sequence information.<n>It also designs an automated assisted assessment scheme based on a large language model.
  arXiv  Detail & Related papers  (2025-07-01T15:10:00Z)
• Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration [63.90193684394165]
  We introduce multi-agent cross-task experiential learning (MAEL), a novel framework that endows LLM-driven agents with explicit cross-task learning and experience accumulation.<n>During the experiential learning phase, we quantify the quality for each step in the task-solving workflow and store the resulting rewards.<n>During inference, agents retrieve high-reward, task-relevant experiences as few-shot examples to enhance the effectiveness of each reasoning step.
  arXiv  Detail & Related papers  (2025-05-29T07:24:37Z)
• Co-Saving: Resource Aware Multi-Agent Collaboration for Software Development [65.94639060883475]
  We propose a resource-aware multi-agent system -- Co-Saving.<n>Our key innovation is the introduction of "shortcuts"<n>Compared to the state-of-the-art MAS ChatDev, our method achieves an average reduction of 50.85% in token usage.
  arXiv  Detail & Related papers  (2025-05-28T02:23:53Z)
• A Novel Zero-Trust Identity Framework for Agentic AI: Decentralized Authentication and Fine-Grained Access Control [7.228060525494563]
  This paper posits the imperative for a novel Agentic AI IAM framework.<n>We propose a comprehensive framework built upon rich, verifiable Agent Identities (IDs)<n>We also explore how Zero-Knowledge Proofs (ZKPs) enable privacy-preserving attribute disclosure and verifiable policy compliance.
  arXiv  Detail & Related papers  (2025-05-25T20:21:55Z)
• AgentOrca: A Dual-System Framework to Evaluate Language Agents on Operational Routine and Constraint Adherence [54.317522790545304]
  We present AgentOrca, a dual-system framework for evaluating language agents' compliance with operational constraints and routines.<n>Our framework encodes action constraints and routines through both natural language prompts for agents and corresponding executable code serving as ground truth for automated verification.<n>Our findings reveal notable performance gaps among state-of-the-art models, with large reasoning models like o1 demonstrating superior compliance while others show significantly lower performance.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• AppAgentX: Evolving GUI Agents as Proficient Smartphone Users [34.70342284525283]
  We propose a novel evolutionary framework for GUI agents that enhances operational efficiency while retaining intelligence and flexibility.<n>Our approach incorporates a memory mechanism that records the agent's task execution history.<n> Experimental results on multiple benchmark tasks demonstrate that our approach significantly outperforms existing methods in both efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-03-04T04:34:09Z)
• CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments [90.29937153770835]
  We introduce CRMArena, a benchmark designed to evaluate AI agents on realistic tasks grounded in professional work environments.<n>We show that state-of-the-art LLM agents succeed in less than 40% of the tasks with ReAct prompting, and less than 55% even with function-calling abilities.<n>Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environments.
  arXiv  Detail & Related papers  (2024-11-04T17:30:51Z)
• Agent-Oriented Planning in Multi-Agent Systems [54.429028104022066]
  We propose AOP, a novel framework for agent-oriented planning in multi-agent systems.<n>In this study, we identify three critical design principles of agent-oriented planning, including solvability, completeness, and non-redundancy.<n> Extensive experiments demonstrate the advancement of AOP in solving real-world problems compared to both single-agent systems and existing planning strategies for multi-agent systems.
  arXiv  Detail & Related papers  (2024-10-03T04:07:51Z)
• Towards Human-Level Understanding of Complex Process Engineering Schematics: A Pedagogical, Introspective Multi-Agent Framework for Open-Domain Question Answering [0.0]
  In the chemical and process industries, Process Flow Diagrams (PFDs) and Piping and Instrumentation Diagrams (P&IDs) are critical for design, construction, and maintenance. Recent advancements in Generative AI have shown promise in understanding and interpreting process diagrams for Visual Question Answering (VQA) We propose a secure, on-premises enterprise solution using a hierarchical, multi-agent Retrieval Augmented Generation (RAG) framework.
  arXiv  Detail & Related papers  (2024-08-24T19:34:04Z)
• Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence [79.5316642687565]
  Existing multi-agent frameworks often struggle with integrating diverse capable third-party agents. We propose the Internet of Agents (IoA), a novel framework that addresses these limitations. IoA introduces an agent integration protocol, an instant-messaging-like architecture design, and dynamic mechanisms for agent teaming and conversation flow control.
  arXiv  Detail & Related papers  (2024-07-09T17:33:24Z)
• BMW Agents -- A Framework For Task Automation Through Multi-Agent Collaboration [0.0]
  We focus on designing a flexible agent engineering framework capable of handling complex use case applications across various domains. The proposed framework provides reliability in industrial applications and presents techniques to ensure a scalable, flexible, and collaborative workflow for multiple autonomous agents.
  arXiv  Detail & Related papers  (2024-06-28T16:39:20Z)
• Experiential Co-Learning of Software-Developing Agents [83.34027623428096]
  Large language models (LLMs) have brought significant changes to various domains, especially in software development. We introduce Experiential Co-Learning, a novel LLM-agent learning framework. Experiments demonstrate that the framework enables agents to tackle unseen software-developing tasks more effectively.
  arXiv  Detail & Related papers  (2023-12-28T13:50:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.