Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents

• URL: http://arxiv.org/abs/2606.13174v1
• Date: Thu, 11 Jun 2026 10:43:40 GMT
• Title: Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents
• Abstract summary: Test-time Rule Acquisition and Compiled Enforcement (TRACE) is a skill-layer pipeline for coding-agent runtimes.<n>Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections.
• Score: 99.67681154381803
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching or exceeding the strongest memory baseline on task pass. These results suggest that compiling corrections into runtime enforcement can address a repeated-friction failure mode that memory alone does not reliably solve, reducing the need for users to restate the same correction across future sessions. Experiment code is available at https://github.com/YujunZhou/TRACE_exp, and the deployable skill is available at https://github.com/YujunZhou/tellonce.

Related papers

• ExecCritic: Learn to Test, Test to Improve for Coding Agents [68.42713285082912]
  Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue.<n>We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it.<n>The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests.
  arXiv  Detail & Related papers  (2026-09-08T17:53:37Z)
• ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts [0.0]
  Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes.<n>We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw.
  arXiv  Detail & Related papers  (2026-08-23T17:09:02Z)
• When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents [62.10623192614306]
  Extracting skills from past successes is critical for the efficient evolution of Large Language Model (LLM) agents.<n>We propose textbfBoundary-Aware Skill Memory (BASM), which augments each skill with explicit boundary fields.<n>BASM consistently outperforms success-distilled skill-memory baselines.
  arXiv  Detail & Related papers  (2026-08-23T10:16:06Z)
• TTHE: Test-Time Harness Evolution [50.26245555541721]
  Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time.<n>We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs.<n>During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces.<n>A judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs.
  arXiv  Detail & Related papers  (2026-07-09T05:53:39Z)
• SWE-Together: Evaluating Coding Agents in Interactive User Sessions [19.069719245971786]
  We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions.<n>We curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes.<n>We build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it.
  arXiv  Detail & Related papers  (2026-06-29T08:35:15Z)
• SKILL.nb: Selective Formalization and Gated Execution for Durable Agent Workflows [16.693609667845948]
  We introduce SKILL.nb, a framework for governing reusable agent with evidence-versus-calibrated lifecycle policies.<n>SKILL.nb uses selective formalization: execution decides which workflow steps should become executable code.<n> gate-conditioned execution lets each step run code when its gates validate, or fall back locally when drift invalidates the executable realization.
  arXiv  Detail & Related papers  (2026-06-06T08:27:18Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• When Should an AI Workflow Release? Always-Valid Inference for Black-Box Generate-Verify Systems [2.805986764620217]
  LLM-enabled AI increasingly produce outputs through iterative generate-evaluate-revise loops.<n>This raises a statistical challenge because deployment-time evaluator scores are adaptively generated and repeatedly monitored.<n>We propose an always-valid release wrapper for existing generator-evaluator pipelines.
  arXiv  Detail & Related papers  (2026-05-13T03:30:39Z)
• Feedback-Normalized Developer Memory for Reinforcement-Learning Coding Agents: A Safety-Gated MCP Architecture [0.0]
  This paper presents RL Developer Memory, a local-first, Model Context Protocol (MCP)-native developer-memory architecture for RL coding agents.<n>It treats memory selection as a logged contextual decision process.<n>System is evaluated on a deterministic 200-case benchmark with RL algorithm bugs, hard negatives, review-gated RL/control cases, and low-risk failures.
  arXiv  Detail & Related papers  (2026-05-02T18:37:36Z)
• TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis [0.0]
  TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct.<n>This paper presents TDAD, an open-source tool that performs pre-change impact analysis for AI coding agents.
  arXiv  Detail & Related papers  (2026-03-18T17:38:22Z)
• ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration [68.89572566071575]
  ETAgent is a training framework for calibrating agent's tool-use behavior.<n>It is designed to progressively calibrate erroneous behavioral patterns to optimal behaviors.
  arXiv  Detail & Related papers  (2026-01-11T11:05:26Z)
• Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clustering [0.0]
  We present functional clustering, a black-box wrapper that eliminates nearly all hallucination-induced errors while providing a tunable confidence score.<n>Our verifier preserves baseline pass@1 on solvable tasks yet slashes the error rate of returned answers from 65% to 2%.<n>Because the method requires only sampling and sandbox execution, it applies unchanged to closed-source APIs and future models.
  arXiv  Detail & Related papers  (2025-05-16T18:19:38Z)
• STAMP: Outlier-Aware Test-Time Adaptation with Stable Memory Replay [76.06127233986663]
  Test-time adaptation (TTA) aims to address the distribution shift between the training and test data with only unlabeled data at test time. This paper pays attention to the problem that conducts both sample recognition and outlier rejection during inference while outliers exist. We propose a new approach called STAble Memory rePlay (STAMP), which performs optimization over a stable memory bank instead of the risky mini-batch.
  arXiv  Detail & Related papers  (2024-07-22T16:25:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.