Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents

• URL: http://arxiv.org/abs/2605.18332v1
• Date: Mon, 18 May 2026 12:49:18 GMT
• Title: Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents
• Abstract summary: Behavioral studies of software engineering agents extract operational rules about which trajectory shapes correlate with higher resolution rates.<n>Each rule is typically derived from a single framework, and whether it transfers, in sign as well as magnitude, to structurally different agent designs has not been directly tested.<n>We separate framework effects from LLM effects by holding each layer fixed in turn, then measure one behavior-outcome effect per configuration.
• Score: 28.650582699234402
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Behavioral studies of LLM-based software engineering agents extract operational rules about which trajectory shapes correlate with higher resolution rates: that a test step follows a code modification, that error cascades are short, or that trajectories are compact. Each rule is typically derived from a single framework, and whether it transfers, in sign as well as magnitude, to structurally different agent designs has not been directly tested. We address this at ecosystem scale: 64,380 SWE-bench runs from 126 agent configurations spanning 43 frameworks, where each configuration pairs an LLM with a framework (e.g., SWE-Agent, OpenHands) that supplies its tools and workflow. We separate framework effects from LLM effects by holding each layer fixed in turn, then measure one behavior-outcome effect per configuration and examine how those effects agree or disagree. Swapping the framework while the LLM is held fixed produces large behavioral differences in every action feature. On most signals, configurations disagree not merely in magnitude but in direction. Error rate is the cleanest case: 47 configurations resolve more issues when their error rate is lower, while 48 resolve more when it is higher. Five other continuous features and three of seven binary patterns from prior SE literature show similar directional disagreement. Framework identity accounts for more of this variation than LLM family: for mean turns, framework explains 64% of the between-configuration variance against the LLM's 10%. The implication is that the same observable behavioral signal can carry opposite meaning for different agent configurations. Behavioral findings from any single framework therefore warrant cross-configuration validation before being claimed as general.

Related papers

• Loop-Back Authority in LLM Agent Teams: A Paired Experiment on Flat and Hierarchical Coordination [0.09999629695552194]
  Hierarchical orchestration, in which a Manager agent reviews worker output and can send it back for revision, is the default coordination pattern in production multi-agent LLM frameworks.<n>We present a paired experiment that holds five LLM agents, their roles, prompts, tools, models, and data fixed and varies one link: whether the Manager may reject a worker's output and oblige a revision.<n>Across 43 paired products and 86 runs of a business-intelligence reporting task, a five-model judge panel and a deterministic specification check score every report.<n>The flat organization scores higher on Utility (d = 0.42, p = 0.009) and
  arXiv  Detail & Related papers  (2026-09-13T19:58:54Z)
• Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling [65.03665015319457]
  We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two modes: action-class miscalibration and action-execution failure.<n>We validate it on a panel of tool-calling models across multiple multi-turn benchmarks.
  arXiv  Detail & Related papers  (2026-09-01T09:08:15Z)
• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• Right-Sizing LLM-Agent Decomposition in VAT Determination: A Pilot Controlled Sweep [0.3384279376065155]
  This pilot studies that choice on bounded cross-border VAT determination with reverse charge.<n>We hold the activity surface fixed (subtasks, tools, I/O schemas, validation checks, orchestrator, base model, and merge policy)<n>The program spans 4,400 runs: a 40-case, five-repeat main sweep, matched-token arms separating prompt-budget from agent-count effects, and three failure-injection arms.
  arXiv  Detail & Related papers  (2026-08-24T15:40:15Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval [0.0]
  We compare six widely-used multi-agent configurations across HumanEval tasks.<n>We find that the leanest architectures match or beat the heaviest on accuracy.
  arXiv  Detail & Related papers  (2026-05-29T19:34:15Z)
• Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables [0.0]
  coding agents read configuration files at session start and are expected to follow the conventions inside them.<n>We report a systematic factorial study of these choices using four manipulated variables.<n>None of the four structural variables or three two-way interactions produces a detectable contrast after multiple-testing correction.
  arXiv  Detail & Related papers  (2026-05-11T06:09:47Z)
• Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure [0.0]
  Coding agents represent a new paradigm in automated software engineering, combining the reasoning capabilities of Large Language Models with tool-augmented interaction loops.<n>Top-ranked LLM-based coding agents still fail on over 20% of benchmarked problems.<n>We present a large-scale empirical study analyzing 9,374 trajectories from 19 agents on 500 tasks.
  arXiv  Detail & Related papers  (2026-04-02T21:56:23Z)
• SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization [6.086377855676357]
  We present SemLoc, a fault localization framework based on structured semantic runtime.<n>We evaluate SemLoc on SemFault-250, a corpus of 250 Python programs with single semantic faults.
  arXiv  Detail & Related papers  (2026-03-31T00:56:43Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• Self-Abstraction from Grounded Experience for Plan-Guided Policy Refinement [61.35824395228412]
  Large language model (LLM) based agents are increasingly used to tackle software engineering tasks.<n>We propose Self-Abstraction from Grounded Experience (SAGE), a framework that enables agents to learn from their own task executions.
  arXiv  Detail & Related papers  (2025-11-08T08:49:38Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• StaAgent: An Agentic Framework for Testing Static Analyzers [7.951459111292028]
  StaAgent is an agentic framework that harnesses the generative capabilities of Large Language Models (LLMs) to systematically evaluate static analyzer rules.<n>StaAgent helps uncover flaws in rule implementations by revealing inconsistent behaviors.<n>We evaluated StaAgent with five state-of-the-art LLMs across five widely used static analyzers.
  arXiv  Detail & Related papers  (2025-07-20T13:41:02Z)
• A Unified Debugging Approach via LLM-Based Multi-Agent Synergy [39.11825182386288]
  FixAgent is an end-to-end framework for unified debug through multi-agent synergy. It significantly outperforms state-of-the-art repair methods, fixing 1.25$times$ to 2.56$times$ bugs on the repo-level benchmark, Defects4J.
  arXiv  Detail & Related papers  (2024-04-26T04:55:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.