RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

• URL: http://arxiv.org/abs/2607.06411v1
• Date: Tue, 07 Jul 2026 15:41:22 GMT
• Title: RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
• Abstract summary: RuBench 1.0 is a benchmark of 25 tasks mined from recent fix commits in five live open-source repositories.<n>Each task is specified in Russian, written from scratch in the style of an actual customer request, not translated.<n>All 25 fix commits postdate the training-data cutoffs of every evaluated model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue. Existing repository-level agentic benchmarks do not measure this setting: their task statements are English by design. We introduce RuBench 1.0, a benchmark of 25 tasks mined from recent fix commits in five live open-source repositories (aiohttp, aiogram, Laravel, NestJS, Fastify; Python, PHP, TypeScript, JavaScript), where each task is specified natively in Russian -- written from scratch in the style of an actual customer request, not translated -- and judged by the upstream maintainer's regression tests, which we withhold from release. All 25 fix commits postdate the training-data cutoffs of every evaluated model, giving a contamination argument that holds task-by-task. We evaluate deployed product configurations (CLI agent + model + reasoning effort) -- Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, and Codex CLI with GPT-5.5 -- with three independent runs each, reporting pass@1 with task-level confidence intervals, paired comparisons, dollar cost, and token usage. The best configuration resolves 78.7% of tasks; at N=25 only the gaps to the weakest model are statistically resolvable, which we state explicitly. Auditing full trajectories of a fifth, hors-concours configuration (Claude Code + Fable 5, July 2, 2026 release), we caught the product silently substituting the model: on 5 of 25 tasks (20%) an official safeguard fallback re-routed routine HTTP-protocol fixes to Opus 4.8 -- direct, reproducible evidence that the deployed product, not the model, is the unit actually measured. We release task statements, metadata, full agent trajectories, and diffs; grading oracles are withheld, with a SHA-256 manifest committed at publication time.

Related papers

• APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows [23.621239974604205]
  APIFlow-Bench is a fully auditable benchmark for long-horizon dependent REST-API performance.<n>We generate synthetic API worlds forward, subtask by subtask.<n>We release all answer keys and 44,362 unredacted execution transcripts.
  arXiv  Detail & Related papers  (2026-08-29T08:12:11Z)
• A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents [16.4855244145731]
  SWE-RPG is a repository-level benchmark that combines executable patch evaluation with validated ground-truth references.<n>SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions.<n>We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends.
  arXiv  Detail & Related papers  (2026-08-10T03:22:04Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks [1.7623000586936592]
  DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.<n>Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream.<n>Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code.
  arXiv  Detail & Related papers  (2026-07-08T21:45:34Z)
• Push Your Agent: Measuring and Enforcing Quantitative Goal Persistence in Long-Horizon LLM Agents [7.058932007120959]
  Long-horizon language agents can make many plausible local tool calls yet fail to persist until a requested count is actually complete.<n>We study this gap as Quantitative Goal Persistence (QGP): whether an agent keeps working until an external verifier confirms enough distinct valid items.<n>PushBench turns this into a benchmark for repository-artifact collection and verifier-backed work units.
  arXiv  Detail & Related papers  (2026-05-22T12:44:01Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python [2.7324157162184157]
  We present a methodology for LLM-assisted continuous code translation in which a large language model translates a production Rust objective into Python.<n>We demonstrate that the Python port resolves 59/80 SWE-bench Verified tasks (73.8%) versus Rust's 56/80 (70.0%), confirming near-parity on real-world agentic tasks.<n>Our evaluation shows that for LLM-based agents where API latency dominates, Python's expressiveness yields a 15.9x code reduction with negligible performance cost.
  arXiv  Detail & Related papers  (2026-04-13T14:21:44Z)
• ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces [17.202580606345666]
  Large language model (LLM) agents are increasingly deployed to automate productivity tasks.<n>Existing benchmarks rely on simplified environments and fail to capture realistic, multi-service environments.<n>We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings.
  arXiv  Detail & Related papers  (2026-04-06T21:09:06Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.