SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

• URL: http://arxiv.org/abs/2608.02499v1
• Date: Mon, 03 Aug 2026 17:03:19 GMT
• Title: SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
• Abstract summary: Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task.<n>We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits.<n>We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE.
• Score: 27.67528029348664
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.

Related papers

• SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents [63.65682461568621]
  We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness.<n>SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints.<n>Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance.
  arXiv  Detail & Related papers  (2026-09-03T17:53:34Z)
• Workspace Topology as an Attack Vector in Agentic Coding Assistants [33.84924547762174]
  Agentic coding assistants are finding widespread use, not just in new code development but in quickly ingesting and leveraging third-party code.<n>This opens up a risk of malicious code being ingested as these coding tools operate with broad access inside developer domains.<n>We study the impact of different dimensions of a novel attack surface on the attack success rate of adversarial prompt injection attempts.
  arXiv  Detail & Related papers  (2026-08-14T20:30:36Z)
• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks [1.7623000586936592]
  DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.<n>Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream.<n>Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code.
  arXiv  Detail & Related papers  (2026-07-08T21:45:34Z)
• SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions [3.67653051806469]
  SWE-Interact places agents in a realistic developer workflow.<n>It starts with vague or incomplete instructions, progressively reveals requirements.<n>It provides targeted feedback, revisions, and new constraints until the full task goal has been handed off.
  arXiv  Detail & Related papers  (2026-06-29T17:17:45Z)
• SWE-Together: Evaluating Coding Agents in Interactive User Sessions [19.069719245971786]
  We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions.<n>We curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes.<n>We build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it.
  arXiv  Detail & Related papers  (2026-06-29T08:35:15Z)
• Code as Agent Harness [107.31925305395957]
  In emerging agentic systems, code is no longer only a target output.<n>Code increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification.<n>This survey provides a unified roadmap toward executable, verifiable, and stateful AI agent systems.
  arXiv  Detail & Related papers  (2026-05-18T17:59:03Z)
• Improving the Efficiency of Language Agent Teams with Adaptive Task Graphs [52.26652574704317]
  Large language models (LLMs) are increasingly deployed in teams, yet existing coordination approaches often occupy two extremes.<n>We introduce Language Agent Teams for Task Evolution (LATTE), a framework for coordinating LLM teams inspired by distributed systems.
  arXiv  Detail & Related papers  (2026-05-07T14:19:17Z)
• SWE-Edit: Rethinking Code Editing for Efficient SWE-Agent [17.74253780030394]
  SWE-Edit decomposes code editing into two specialized subagents: a Viewer that extracts task-relevant code on demand, and an Editor that executes modifications from high-level plans.<n>On SWE-bench Verified, SWE-Edit improves resolved rate by 2.1% while reducing inference cost by 17.9%.
  arXiv  Detail & Related papers  (2026-04-28T20:35:09Z)
• SWE-Adept: An LLM-Based Agentic Framework for Deep Codebase Analysis and Structured Issue Resolution [9.339988760379915]
  Large language models (LLMs) exhibit strong performance on self-contained tasks.<n>SWE-Adept is a framework where a localization agent identifies issue-relevant code locations and a resolution agent implements the corresponding fixes.<n> Experiments on SWE-Bench Lite and SWE-Bench Pro demonstrate that SWE-Adept consistently outperforms prior approaches in both issue localization and resolution.
  arXiv  Detail & Related papers  (2026-03-01T23:52:30Z)
• When Disagreements Elicit Robustness: Investigating Self-Repair Capabilities under LLM Multi-Agent Disagreements [56.29265568399648]
  We argue that disagreements prevent premature consensus and expand the explored solution space.<n>Disagreements on task-critical steps can derail collaboration depending on the topology of solution paths.
  arXiv  Detail & Related papers  (2025-02-21T02:24:43Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.