Related papers: Agent trajectories as programs: fingerprinting and programming coding-agent behavior

Agent trajectories as programs: fingerprinting and programming coding-agent behavior

URL: http://arxiv.org/abs/2606.16988v1
Date: Mon, 15 Jun 2026 17:28:41 GMT
Title: Agent trajectories as programs: fingerprinting and programming coding-agent behavior
Abstract summary: Benchmark scores tell you what an agent got right; they do not tell you how it got there.<n>In this work, we introduce methods for comparing agents procedurally in different contexts, where the model, tasks, and approaches vary.<n>A probe over these procedural signatures attributes an unseen trajectory to the correct agent at 85.7% accuracy, controlling for leakage across tasks.
Score: 0.49316866264940024
License: http://creativecommons.org/licenses/by-sa/4.0/
Abstract: Benchmark scores tell you what an agent got right; they do not tell you how it got there. In this work, we introduce methods for comparing agents procedurally in different contexts, where the model, tasks, and approaches vary. We compare ten agents and find that they are identifiable by their behavioral habits, which we define as fingerprints: a probe over these procedural signatures attributes an unseen trajectory to the correct agent at 85.7% accuracy, controlling for leakage across tasks. We develop procedural representations for agent problem-solving procedures with an emergent vocabulary induction technique that is meant to be maximally compressive to avoid surface-level variation while being expressive enough to unveil the quirks of the models' patterns. We apply our framework to the software engineering evaluation dataset SWE-Bench to study the structural distinctness of agent trajectories and find that behavior is most similar between models from similar release periods and those that are distilled from one another (e.g., a distilled student model and its teacher have a Jensen-Shannon divergence of 0.25, about half the distance between other model pairs). As more models saturate evaluations, we believe that it will be important to probe model behavior along more holistic dimensions than success rates alone. We introduce ProcGrep, a library for auditing and evaluating agents for how they approach tasks at a procedural level given their traces in a top-down fashion. We believe this work has a range of applications to help developers work with and program coding agents, such as task-aware model routing, agent monitoring, and finer-grained cost analysis.

Related papers

An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery [4.502708670326481]
Large language model coding agents increasingly perform open-ended data modeling and analysis.<n>We propose an experimental design and analysis framework for evaluating this discovery process.<n>We develop a utility-aligned canonical decomposition to characterize the dominant direction of the reasoning-effort effect.
arXiv Detail & Related papers (2026-07-07T15:43:22Z)
Latent Programming Horizons in Coding Agents [6.8463578914435645]
coding agent solving a software-engineering task spends dozens of steps reasoning, editing code, and running tests.<n>We show that the residual streams of language models under coding agents linearly encode properties of the evolving program.<n> Probes trained to predict the outcome of future edits achieve performance above chance up to roughly 25 steps in advance.
arXiv Detail & Related papers (2026-07-06T15:08:26Z)
Dissecting model behavior through agent trajectories [15.811597127775812]
We formalize the intent-execution' gap: the mismatch between what the model intends and what the harness executes, and vice versa.<n>To illustrate the impact of this harness-model alignment, we develop a simple and customizable harness called Simple Strands Agent' (SSA)<n>We make two contributions: (i) we reproduce or improve on the pass@1 performance reported by diverse model-provider families on popular agentic benchmarks, and (ii) building on an analysis of 138k trajectories generated by SSA.
arXiv Detail & Related papers (2026-06-16T03:17:03Z)
How to Interpret Agent Behavior [56.59836196946289]
We introduce ACT*ONOMY, a taxonomy for describing and analyzing agent behavior at runtime.<n>By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently.
arXiv Detail & Related papers (2026-05-13T14:52:40Z)
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
arXiv Detail & Related papers (2026-03-15T16:13:58Z)
AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent [57.10083973844841]
AgentArk is a novel framework to distill multi-agent dynamics into the weights of a single model.<n>We investigate three hierarchical distillation strategies across various models, tasks, scaling, and scenarios.<n>By shifting the burden of computation from inference to training, the distilled models preserve the efficiency of one agent while exhibiting strong reasoning and self-correction performance of multiple agents.
arXiv Detail & Related papers (2026-02-03T19:18:28Z)
AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress [71.02263260394261]
Large language models (LLMs) still encounter challenges in multi-turn decision-making tasks.<n>We build process reward models (PRMs) to evaluate each decision and guide the agent's decision-making process.<n>AgentPRM captures both the interdependence between sequential decisions and their contribution to the final goal.
arXiv Detail & Related papers (2025-11-11T14:57:54Z)
Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories [10.751728274263536]
This paper presents an empirical study of agent trajectories, namely the execution traces capturing the steps agents take when attempting to resolve software issues.<n>We analyse trajectories from three state-of-the-art code agents (OpenHands, SWE-agent, and Prometheus) on the SWE-Bench benchmark, examining both successful and failed attempts.
arXiv Detail & Related papers (2025-10-31T18:58:13Z)
Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
arXiv Detail & Related papers (2025-10-02T06:25:10Z)
Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark [72.46357004059661]
Generalist Virtual Agents (GVAs) have shown significant promise in autonomous task execution.<n>To address these challenges, we propose Similar, a Step-Wise Multi-Dimensional Generalist Reward Model.<n>Similar offers fine-grained signals for agent training and can choose better action for inference-time scaling.
arXiv Detail & Related papers (2025-03-24T13:30:47Z)
Interpretable Interaction Modeling for Trajectory Prediction via Agent Selection and Physical Coefficient [1.6954753390775528]
We present ASPILin, which manually selects interacting agents and replaces the attention scores in Transformer with a newly computed physical correlation coefficient.<n>Surprisingly, these simple modifications can significantly improve prediction performance and substantially reduce computational costs.
arXiv Detail & Related papers (2024-05-21T18:45:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.