TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework

• URL: http://arxiv.org/abs/2606.05570v1
• Date: Thu, 04 Jun 2026 01:42:40 GMT
• Title: TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework
• Abstract summary: CohenBench is a benchmark for tasks on an open-source compiler-based framework.<n>It grades each run by applying the agent's patch and running the framework's test suite.<n>We evaluate seven coding agents spanning three frontier model families and one open-weight model.
• Score: 0.9012337166501982
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Repository-level coding benchmarks face a trade-off between task difficulty and evaluation reliability: tasks that challenge frontier models often involve large codebases with incomplete test coverage, while human review does not scale. We introduce TensorBench, a benchmark of 199 feature-addition and refactoring tasks on an open-source compiler-based tensor framework that extends PyTorch with first-class support for dense and sparse tensors. Tasks cover new sparse formats, dense optimization passes, IR transformations, scheduler changes, runtime components, and high-level numerical operators. TensorBench grades each run by applying the agent's patch and running the framework's test suite, which includes the pre-existing randomized regression tests and any tests the agent adds. For feature-addition tasks, a pass means that the patched repository preserves the tested pre-existing behavior and satisfies the agent-added checks for the requested feature. We evaluate seven coding agents spanning three frontier model families and one open-weight model. Pass rates under this criterion range from $64.8\%$ for the strongest agent to $22.1\%$ for the weakest. Agents pass different subsets of tasks: pairwise Cohen's $κ$ ranges from $-0.07$ to $0.43$, with $κ= 0.05$ for the two strongest agents.

Related papers

• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• A Jagged Frontier: Evaluating Robustness of Code Agents to Semantics-Preserving Transformations [7.48361039542955]
  We evaluate whether coding agents that repair repository-level issues remain reliable when the surrounding code variations is rewritten into a semantically equivalent form.<n>We introduce a random variant sampler that applies common semantics-preserving transformations (SPTs)<n>We evaluate two agentic scaffolds (mini-SWE agent and OpenCode) each backed by one of four frontier models.
  arXiv  Detail & Related papers  (2026-08-18T23:46:11Z)
• Grounding AI Agents in Contracts: An Empirical Evaluation of Spec-Driven Test Generation [4.061618191574875]
  Spec-Driven Test Generation instructs an agent to first reason about -- and explicitly document -- code pre-conditions, post-conditions, and undefined behaviors.<n>This intermediate semi-formal specification acts as a cognitive scaffold to guide subsequent test generation.
  arXiv  Detail & Related papers  (2026-08-17T22:36:12Z)
• SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring [31.684059339633013]
  SWE-Bench ProMax is an expert-curated, multilingual code benchmark of 170 instances drawn from real commits.<n>Every instance undergoes rigorous, multi-stage curation that addresses quality problems identified in prior benchmarks.<n>Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate.
  arXiv  Detail & Related papers  (2026-08-10T16:23:19Z)
• SSTQ:Privacy-Preserving Vector Quantization via Subsampled Stochastic TurboQuant [79.24089819400126]
  Subsampled TurboQuant (SSTQ) is a framework that combines overcomplete equal-width tight frames, coordinate subsampling, and privacy-aware quantization.<n>SSTQ achieves mean squared error scaling while using only $lceil log N il + b$ bits per client, where $N = (d)$ is the frame size.<n>We also derive a surrogate privacy-aware codebook objective that reduces the codebook-dependent MSE scaling from $O(4b)$ to $O(2b)$.
  arXiv  Detail & Related papers  (2026-08-05T17:51:25Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• ExecuGraph: A Multi-Agent, Execution-Grounded Framework for Reliable Backend Code Synthesis with Large Language Models [2.3004655342211078]
  Large Language Models generate plausible backend code, but a single-pass paradigm provides no guarantee of correctness or runtime reliability.<n>We present ExecuGraph, a multi-agent framework that places execution-based validation at the center of backend code.<n>A per-agent ablation, retry-budget sweep, error-class taxonomy, and test-source audit are reported.
  arXiv  Detail & Related papers  (2026-06-18T05:58:17Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks [25.713629634281077]
  We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage.<n> TASTE selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through difficulty evolution.<n>Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability.
  arXiv  Detail & Related papers  (2026-05-27T14:45:59Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning [14.185814237633958]
  Descent-Guided Policy Gradient (DG-PG) is a framework that constructs noise-free per-agent guidance gradients.<n>We prove that DG-PG reduces gradient variance from $(N)$ to $mathcalO(1)$, preserves the equilibria of the cooperative game, and achieves agent-independent sample complexity.
  arXiv  Detail & Related papers  (2026-02-23T17:45:08Z)
• ATTest: Agent-Driven Tensor Testing for Deep Learning Library Modules [19.355376741404267]
  Unit testing of Deep Learning (DL) libraries is challenging due to complex numerical semantics and implicit tensor constraints.<n>This paper proposes ATTest, an agent-driven testing framework for module-level unit test generation.
  arXiv  Detail & Related papers  (2026-02-15T04:47:58Z)
• OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding [57.39403818250357]
  We introduce OctoBench, which benchmarks scaffold-aware instruction following in repository-grounded agentic coding.<n> OctoBench includes 34 environments and 217 tasks instantiated under three scaffold types, and is paired with 7,098 objective checklist items.<n>Experiments reveal a systematic gap between task-solving and scaffold-aware compliance, underscoring the need for training and evaluation.
  arXiv  Detail & Related papers  (2026-01-15T12:36:08Z)
• Agentic Rubrics as Contextual Verifiers for SWE Agents [8.469998524915818]
  We show that Agentic rubrics provide an efficient, scalable, and granular verification signal for SWE agents.<n>Results show that Agentic rubrics are consistent with ground-truth tests while also flagging issues that tests do not capture.
  arXiv  Detail & Related papers  (2026-01-07T18:38:23Z)
• On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents [58.79302663733703]
  Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents.<n>The impact of clumsy or even malicious agents--those who frequently make errors in their tasks--on the overall performance of the system remains underexplored.<n>This paper investigates what is the resilience of various system structures under faulty agents on different downstream tasks.
  arXiv  Detail & Related papers  (2024-08-02T03:25:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.