Agent Mentor: Framing Agent Knowledge through Semantic Trajectory Analysis

• URL: http://arxiv.org/abs/2604.10513v1
• Date: Sun, 12 Apr 2026 08:02:54 GMT
• Title: Agent Mentor: Framing Agent Knowledge through Semantic Trajectory Analysis
• Abstract summary: We introduce an analytics pipeline implemented as part of the Agent Mentor open-source library.<n>It monitors and incrementally adapts the system prompts defining another agent's behavior.<n>The pipeline improves performance by systematically injecting corrective instructions into the agent's knowledge.
• Score: 4.365760422569902
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agent development relies heavily on natural language prompting to define agents' tasks, knowledge, and goals. These prompts are interpreted by Large Language Models (LLMs), which govern agent behavior. Consequently, agentic performance is susceptible to variability arising from imprecise or ambiguous prompt formulations. Identifying and correcting such issues requires examining not only the agent's code, but also the internal system prompts generated throughout its execution lifecycle, as reflected in execution logs. In this work, we introduce an analytics pipeline implemented as part of the Agent Mentor open-source library that monitors and incrementally adapts the system prompts defining another agent's behavior. The pipeline improves performance by systematically injecting corrective instructions into the agent's knowledge. We describe its underlying mechanism, with particular emphasis on identifying semantic features associated with undesired behaviors and using them to derive corrective statements. We evaluate the proposed pipeline across three exemplar agent configurations and benchmark tasks using repeated execution runs to assess effectiveness. These experiments provide an initial exploration of automating such a mentoring pipeline within future agentic governance frameworks. Overall, the approach demonstrates consistent and measurable accuracy improvements across diverse configurations, particularly in settings dominated by specification ambiguity. For reproducibility, we released our code as open source under the Agent Mentor library.

Related papers

• The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows [72.9051182568928]
  We present ReASearch, a unified framework for reasoning-driven optimization.<n>With a shared agent loop and domain-specific tools, ReASearch instantiates the exact same scaffold to optimize prompts, programs, and ML.<n>We observe that complex search behaviors emerge naturally from the agent's reasoning process.
  arXiv  Detail & Related papers  (2026-08-07T02:15:39Z)
• How to Interpret Agent Behavior [56.59836196946289]
  We introduce ACT*ONOMY, a taxonomy for describing and analyzing agent behavior at runtime.<n>By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently.
  arXiv  Detail & Related papers  (2026-05-13T14:52:40Z)
• AgentSPEX: An Agent SPecification and EXecution Language [17.439531520677594]
  We introduce AgentSPEX, an agent framework with explicit control flow and modular structure.<n>AgentSPEX supports typed steps, branching and loops, parallel execution, reusable submodules, and explicit state management.<n>We show through a user study that AgentSPEX provides a more interpretable and accessible workflow-authoring paradigm than a popular existing agent framework.
  arXiv  Detail & Related papers  (2026-04-14T23:16:25Z)
• Willful Disobedience: Automatically Detecting Failures in Agentic Traces [2.2365938060262356]
  AgentPex is an AI-powered tool designed to systematically evaluate agentic traces.<n>It extracts behavioral rules from agent prompts and system instructions, then uses these specifications to automatically evaluate traces for compliance.<n>We evaluate AgentPex on 424 traces from 2-bench across models in telecom, retail, and airline customer service.
  arXiv  Detail & Related papers  (2026-03-25T00:33:39Z)
• When Only the Final Text Survives: Implicit Execution Tracing for Multi-Agent Attribution [10.973058523304042]
  We introduce IET (Implicit Execution Tracing), a metadata-independent framework that enables token-level attribution directly from generated text.<n>During generation, agent-specific keyed signals are embedded into the token distribution, transforming the text into a self-describing execution trace detectable only with a secret key.<n>At detection time, a transition-aware scoring method identifies agent handover points and reconstructs the interaction graph.
  arXiv  Detail & Related papers  (2026-03-18T07:34:51Z)
• VeRO: An Evaluation Harness for Agents to Optimize Agents [5.227525836910522]
  We introduce VERO (Versioning, Rewards, and Observations), which provides a reproducible evaluation harness with versioned agent snapshots, budget-controlled evaluation, and structured execution traces.<n>We conduct an empirical study comparing target agents and analyzing which modifications reliably improve target agent performance.
  arXiv  Detail & Related papers  (2026-02-25T23:40:22Z)
• The Why Behind the Action: Unveiling Internal Drivers via Agentic Attribution [63.61358761489141]
  Large Language Model (LLM)-based agents are widely used in real-world applications such as customer service, web navigation, and software engineering.<n>We propose a novel framework for textbfgeneral agentic attribution, designed to identify the internal factors driving agent actions regardless of the task outcome.<n>We validate our framework across a diverse suite of agentic scenarios, including standard tool use and subtle reliability risks like memory-induced bias.
  arXiv  Detail & Related papers  (2026-01-21T15:22:21Z)
• Agentic AI Process Observability: Discovering Behavioral Variability [2.273531916003657]
  AI agents that leverage Large Language Models (LLMs) are increasingly becoming core building blocks of modern software systems.<n> frameworks enable the definition of agent setups using natural language prompting.<n>Within such setups, agent behavior is non-deterministic for any given input.
  arXiv  Detail & Related papers  (2025-05-26T15:26:07Z)
• Agent-as-a-Judge: Evaluate Agents with Agents [61.33974108405561]
  We introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We present DevAI, a new benchmark of 55 realistic automated AI development tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:57:02Z)
• Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement [112.04307762405669]
  G"odel Agent is a self-evolving framework inspired by the G"odel machine.<n>G"odel Agent can achieve continuous self-improvement, surpassing manually crafted agents in performance, efficiency, and generalizability.
  arXiv  Detail & Related papers  (2024-10-06T10:49:40Z)
• Explaining Reinforcement Learning Policies through Counterfactual Trajectories [147.7246109100945]
  A human developer must validate that an RL agent will perform well at test-time. Our method conveys how the agent performs under distribution shifts by showing the agent's behavior across a wider trajectory distribution. In a user study, we demonstrate that our method enables users to score better than baseline methods on one of two agent validation tasks.
  arXiv  Detail & Related papers  (2022-01-29T00:52:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.