Output Format x Model Identity: Interaction Effects in Single-Round Coding Agent Performance

• URL: http://arxiv.org/abs/2607.21674v1
• Date: Thu, 23 Jul 2026 10:18:42 GMT
• Title: Output Format x Model Identity: Interaction Effects in Single-Round Coding Agent Performance
• Abstract summary: We conduct a controlled experiment with 3 models (DeepSeek V4, Doubao 2.0, Qwen 3.7 Max)<n>We identify a distinct failure mechanism -- format misuse where agents execute it with excessive scope.<n>We propose a model-specific strategy, a tooldesign principle that constrains format semantics to the agent's own step.
• Score: 4.342406076796542
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Output format is not a neutral implementation detail -- it can reorder model rankings, amplify or suppress individual model differences, and determine whether a coding agent succeeds or fails. We conducted a controlled single-round experiment with 3 models (DeepSeek V4, Doubao 2.0 Pro, Qwen 3.7 Max) x 3 output formats (full file, JSON Patch, unified diff) x 6 tasks x 20 repetitions, totaling 4,013 runs across 4 open-source projects. Only one project (tqdm) yielded non-zero success rates: dotenv, requests, and jsoup yielded zero successes in 2,551 runs. Our central finding is a format x model interaction with no universally optimal format. Doubao achieves 94% success with JSON Patch (Cohen's h = 1.57, p < 0.001), DeepSeek excels at unified diff (66%, h = 0.63), and Qwen shows a small but significant full-file preference (50%, h = 0.29, p < 0.05). Beyond these headline results, we identify a distinct failure mechanism -- format misuse -- where agents correctly diagnose a problem but execute it with excessive scope, most vividly when a one-line fix is applied as a full-file replacement. We propose a model-specific output strategy, a tool-design principle that constrains format semantics to the agent's own localization step, and release all data and templates for reproducibility. All experimental data are available at https://doi.org/10.5281/zenodo.21505157.

Related papers

• Architecture as Capability Equalizer for Coding Agents [0.8741110159894001]
  LLM-based coding agents generate complete software systems from high-level descriptions.<n>We present a controlled experiment comparing five informationally equivalent specification formats.<n>Across 90 multi-turn agent trials, specification format shows a strong format x model interaction.
  arXiv  Detail & Related papers  (2026-08-22T03:04:12Z)
• Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models [0.0]
  We investigate Parishad, a structured multi-agent system involving five roles, by deploying it on Qwen2.5-7B-Instruct.<n>Our results demonstrate that communication format and implementation details determine outcomes more than architectural complexity.
  arXiv  Detail & Related papers  (2026-07-29T13:53:02Z)
• Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions [0.0]
  Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user.<n>We introduce quantum-api-drift, a benchmark for measuring version fidelity.<n>We instantiate the benchmark with Qiskit, a representative quantum SDK that underwent substantial interface changes across v0.43, v1.3, and v2.0.
  arXiv  Detail & Related papers  (2026-07-05T01:16:12Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Articulate but Wrong: Self-Review Failures in LLM-Based Code Modernization [1.0164694825170502]
  Large language model (LLM) agents are increasingly used to migrate legacy code to modern stacks.<n>We run 1,980 real modernization calls across 11 production LLMs from 7 distinct families on a balanced 60-snippet legacy-Python-2 corpus.<n>We find that semantic-preservation drift is prevalent and sharply separable from a cleanly-controlled baseline.
  arXiv  Detail & Related papers  (2026-05-20T05:00:31Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models [2.064923532131528]
  Deployed language models must produce outputs that are both correct and format-compliant.<n>We study this structured-output reliability gap using two mathematical benchmarks -- GSM8K and MATH.<n>We develop AloLab, an iterative system-prompt requiring only black-box API access to the target model.
  arXiv  Detail & Related papers  (2026-05-04T09:07:44Z)
• Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models [0.0]
  LLMs deployed for natural-language querying of analytical databases suffer from two intertwined failures.<n>We benchmark three frontier LLMs on 100 natural-language questions over the Cleaned Contoso Retail dataset in ClickHouse.
  arXiv  Detail & Related papers  (2026-04-28T02:53:23Z)
• When Career Data Runs Out: Structured Feature Engineering and Signal Limits for Founder Success Prediction [0.0]
  Predicting startup success from founder career data is hard.<n>We engineer 28 structured features directly from raw fields -- jobs, education, exits -- and combine them with a deterministic rule layer and XG boosted stumps.<n>Our model achieves Val F0.5 = 0.3030, Precision = 0.3333, Recall = 0.2222 -- +17.7pp improvement over the zero-shot LLM baseline.
  arXiv  Detail & Related papers  (2026-04-01T00:28:53Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction [95.91743732150233]
  Goedel-Prover-V2, a series of open-source language models, set a new state-of-the-art in automated theorem proving.<n>We generate synthetic tasks of increasing difficulty to train the model to master increasingly complex theorems.<n>Goedel-Prover-V2-32B achieves 88.1% on MiniF2F at pass@32 in standard mode and 90.4% in self-correction mode.
  arXiv  Detail & Related papers  (2025-08-05T16:28:22Z)
• ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis [50.020850767257095]
  We propose ProofAug, a procedure that equips LLMs with automation methods at various granularities.<n>Our method is validated on the miniF2F benchmark using the open-source deep-math-7b-base model and the Isabelle proof assistant.<n>We also implement a Lean 4 version of ProofAug that can improve the pass@1 performance of Kimina-Prover-seek-Distill-1.5B from 44.3% to 50.4%.
  arXiv  Detail & Related papers  (2025-01-30T12:37:06Z)
• APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets [99.8988504388011]
  APIGen is an automated data generation pipeline designed to synthesize verifiable high-quality datasets for function-calling applications. We leverage APIGen and collect 3,673 executable APIs across 21 different categories to generate diverse function-calling datasets. We release a dataset containing 60,000 high-quality entries, aiming to advance the field of function-calling agent domains.
  arXiv  Detail & Related papers  (2024-06-26T17:49:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.