What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?

• URL: http://arxiv.org/abs/2602.01611v1
• Date: Mon, 02 Feb 2026 04:02:03 GMT
• Title: What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?
• Abstract summary: We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces.<n>Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting.<n>We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces.
• Score: 32.72487312750579
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly evaluated as interactive agents, yet standard agent benchmarks conflate two qualitatively distinct sources of success: semantic tool-use and interface-specific interaction pattern memorization. Because both mechanisms can yield identical task success on the original interface, benchmark scores alone are not identifiable evidence of environment-invariant capability. We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces while preserving task semantics and execution behavior. Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that trajectory-SFT substantially amplifies interface shortcutting: trained agents degrade sharply under minimal interface rewrites, while non-trajectory-trained models remain largely stable. We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces, and show that interface shortcutting exhibits environment-dependent, non-monotonic training dynamics that remain invisible under standard evaluation. Our code is available at https://anonymous.4open.science/r/What-Do-Agents-Learn-from-Trajectory-SFT-Semantics-or-Interfaces-- 0831/.

Related papers

• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both [55.182037225013836]
  ATLAS is a framework in which a single discrete 'word', termed as a functional token, serves both as an agentic operation and a latent visual reasoning unit.<n> ATLAS achieves superior performance on challenging benchmarks while maintaining clear interpretability.
  arXiv  Detail & Related papers  (2026-05-14T17:59:55Z)
• Discovering Reinforcement Learning Interfaces with Large Language Models [0.9558392439655014]
  Reinforcement learning systems rely on environment interfaces that specify observations and reward functions.<n>Recent work has automated reward design using large language models (LLMs), but these approaches assume fixed observations.<n>We study RL task interface discovery from raw simulator state, where both observation mappings and reward functions must be generated.
  arXiv  Detail & Related papers  (2026-05-05T06:27:20Z)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction [127.64173950476702]
  We study direct corpus interaction (DCI), where an agent searches the raw corpus directly with general-purpose terminal tools.<n>This approach requires no offline indexing and adapts naturally to evolving local corpora.<n>Across IR benchmarks and end-to-end agentic search tasks, this simple setup substantially outperforms strong sparse, dense, and reranking baselines.
  arXiv  Detail & Related papers  (2026-05-03T19:13:11Z)
• SAGE: A Service Agent Graph-guided Evaluation Benchmark [27.342044311161654]
  We propose SAGE (Service Agent Graph-guided Evaluation), a universal benchmark for automated, dual-axis assessment.<n>SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance.<n>We also observe Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures.
  arXiv  Detail & Related papers  (2026-04-10T12:55:23Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation [0.0]
  Agent-Diff is a benchmarking framework for evaluating agentic Large Language Models (LLMs) on real-world tasks that execute code via external APIs.<n>We provide benchmarks for nine LLMs across 224 tasks utilizing enterprise software.<n>We also evaluate the robustness of the framework with ablation experiments to assess the contribution of access to API documentation on benchmark performance.
  arXiv  Detail & Related papers  (2026-02-11T13:31:18Z)
• AgentSkiller: Scaling Generalist Agent Intelligence through Semantically Integrated Cross-Domain Data Synthesis [30.512393568258105]
  Large Language Model agents demonstrate potential in solving real-world problems via tools, yet generalist intelligence is bottlenecked by scarce high-quality, long-horizon data.<n>We propose AgentSkiller, a fully automated framework synthesizing multi-turn interaction data across realistic, semantically linked domains.
  arXiv  Detail & Related papers  (2026-02-10T03:21:42Z)
• ANCHOR: Branch-Point Data Generation for GUI Agents [52.22377425487]
  End-to-end GUI agents for real desktop environments require large amounts of high-quality interaction data.<n>We present a trajectory expansion framework Anchor that bootstraps scalable desktop supervision from a small set of verified seed demonstrations.<n>Experiments on standard desktop benchmarks, OSWorld and WindowsAgentArena, show that models fine-tuned on our expanded corpus achieve consistent improvements.
  arXiv  Detail & Related papers  (2026-02-06T19:55:26Z)
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting [92.57796055887995]
  We introduce ECHO, a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents.<n> ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts.<n>We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation.
  arXiv  Detail & Related papers  (2025-10-11T18:11:09Z)
• GUI-PRA: Process Reward Agent for GUI Tasks [25.20594694997543]
  Process Reward Models (PRMs) are a promising solution, as they can guide these agents with crucial process signals during inference.<n>PRMs suffer from a "lost in the middle" phenomenon, where the overwhelming historical context compromises the evaluation of the current step.<n>We introduce GUI-PRA (Process Reward Agent for GUI Tasks), a judge agent designed to better provide process reward than standard PRM.
  arXiv  Detail & Related papers  (2025-09-27T11:42:36Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• Generative Interfaces for Language Models [70.25765232527762]
  We propose a paradigm in which large language models (LLMs) respond to user queries by proactively generating user interfaces (UIs)<n>Our framework leverages structured interface-specific representations and iterative refinements to translate user queries into task-specific UIs.<n>Results show that generative interfaces consistently outperform conversational ones, with up to a 72% improvement in human preference.
  arXiv  Detail & Related papers  (2025-08-26T17:43:20Z)
• AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning [82.42421823672954]
  AgentCPM-GUI is built for robust and efficient on-device GUI interaction.<n>Our training pipeline includes grounding-aware pre-training to enhance perception.<n>AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T07:30:29Z)
• Agent-Environment Alignment via Automated Interface Generation [10.469261669159367]
  Large language model (LLM) agents have shown impressive reasoning capabilities in interactive decision-making tasks.<n> mismatches often happen between the internal expectations of the agent regarding the influence of its issued actions and the actual state transitions in the environment.<n>We propose textbfALIGN, an interface that enhances both the static information of the environment and the step-wise observations returned to the agent.
  arXiv  Detail & Related papers  (2025-05-27T11:44:50Z)
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents [58.18100825673032]
  This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions.<n>In the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively)
  arXiv  Detail & Related papers  (2025-01-21T17:48:10Z)
• Interfacing Foundation Models' Embeddings [131.0352288172788]
  We present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval.
  arXiv  Detail & Related papers  (2023-12-12T18:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.