TeleSWEBench: A Commit-Driven Benchmark for Evaluating LLM-Powered Software Engineering in Telecommunications

• URL: http://arxiv.org/abs/2606.05001v1
• Date: Wed, 03 Jun 2026 15:19:40 GMT
• Title: TeleSWEBench: A Commit-Driven Benchmark for Evaluating LLM-Powered Software Engineering in Telecommunications
• Abstract summary: General-purpose coding benchmarks fail to capture the stateful logic and strict requirements of telecommunications.<n>We introduce TeleSWEBench, the first commit-driven benchmark specifically designed to measure an agent's performance in the telecom domain.
• Score: 8.423549171787396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the telecommunications field embracing zero touch management alongside novel O-RAN and AI-RAN frameworks, contemporary telecom networks now function as immensely intricate and heavily softwareized codebases. While automated software engineering (ASE) tools and Software Engineering (SWE) Agents hold the potential to alleviate the critical code generation bottleneck in this domain, their ability to navigate and modify specialized, mathematically rigorous wireless stacks like srsRAN 5G remains unverified. General-purpose coding benchmarks fail to capture the stateful logic and strict requirements of telecommunications, leaving a critical evaluation gap. In this paper, we introduce TeleSWEBench, the first commit-driven benchmark specifically designed to measure an agent's performance in the telecom domain. We mine real developer commits from the srsRAN 5G repository and distill them into structured test cases across three difficulty tiers (Easy, Medium, and Difficult). Our benchmark consists of 734 questions that are accompanied by executable unit tests. To avoid the rigidity of test cases, we further propose a hierarchical LLM as a Judge framework called TeleJudge that scores agent outputs at the file level and aggregates verdicts holistically. This follows an evaluation based on context and semantic similarity in parallel to a standard unit test-based evaluation. Using this benchmark, we evaluate AIDER, OpenHands, and the ClaudeCode frameworks, powered by state-of-the-art reasoning LLMs, including Qwen3, GPT OSS, Gemma 4, Kimi, and Qwencoder 2.5. Our two-stage evaluation reveals that models suffer from a lack of both localization accuracy and functional correctness, with the strongest ASE tools achieving up to 25% of shippable changes.

Related papers

• SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents [63.65682461568621]
  We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness.<n>SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints.<n>Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance.
  arXiv  Detail & Related papers  (2026-09-03T17:53:34Z)
• RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents [0.0]
  RigorBench is the first benchmark to measure process discipline in AI coding agents.<n>It evaluates harnesses across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity.<n>Our results show that structured process discipline improves process quality scores by an average of 41% and raises downstream outcome correctness by 17%.
  arXiv  Detail & Related papers  (2026-06-21T21:41:34Z)
• TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications? [12.347241408492279]
  We present TeleCom-Bench, a comprehensive benchmark comprising 12 evaluation sets with 22,678 curated samples, which evaluates Large Language Models.<n>We show that current LLMs function competently as diagnosticians but fail as field engineers.<n>This capability gap demonstrates that current LLMs function competently as diagnosticians but fail as field engineers.
  arXiv  Detail & Related papers  (2026-05-18T08:14:49Z)
• SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle [46.833916047516716]
  SWE-Cycle is a benchmark of 489 rigorously filtered instances of autonomous code agents.<n>SWE-Judge combines static code review with dynamic testing to accurately verify functional correctness.<n>The results reveal a sharp drop in solve rates when transitioning from isolated tasks to FullCycle execution.
  arXiv  Detail & Related papers  (2026-05-13T08:05:16Z)
• TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications [2.2508462342902633]
  Large language models (LLMs) are increasingly deployed in the telecommunications domain for critical tasks.<n>We introduce TeleEmbedBench, the first large-scale, multi-corpus embedding benchmark designed specifically for telecommunications.
  arXiv  Detail & Related papers  (2026-04-20T04:00:13Z)
• ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS [12.977178538993806]
  The HarmonyOS ecosystem relies heavily on ArkTS, a statically typed extension of TypeScript.<n>Despite its growing importance, the ecosystem lacks robust tools for automated code repair.<n>We present ArkEval, a unified framework for ArkTS automated repair workflow evaluation and benchmark construction.
  arXiv  Detail & Related papers  (2026-02-09T16:28:29Z)
• Can Large Language Models Detect Real-World Android Software Compliance Violations? [3.7278558081099544]
  Current models struggle to detect compliance violations in Android applications across diverse legal frameworks.<n>We propose emphCompliBench, a novel evaluation framework for assessing LLMs' ability to detect compliance violations.<n>Experiments with six models, including GPT-4O and Claude-3.5, show emphCompliBench improves compliance detection.
  arXiv  Detail & Related papers  (2025-11-01T16:58:33Z)
• Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol [83.83217247686402]
  Large Language Models (LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions.<n>Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance.<n>This paper decomposes LLM applications into a three-layer architecture: textbftextitSystem Shell Layer, textbftextitPrompt Orchestration Layer, and textbftextitLLM Inference Core.
  arXiv  Detail & Related papers  (2025-08-28T13:00:28Z)
• ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks [64.86209459039313]
  ThinkGeo is an agentic benchmark designed to evaluate tool-augmented agents on remote sensing tasks via structured tool use and multi-step planning.<n>We implement a ReAct-style interaction loop and evaluate both open and closed-source LLMs on 486 structured agentic tasks with 1,773 expert-verified reasoning steps.<n>Our analysis reveals notable disparities in tool accuracy and planning consistency across models.
  arXiv  Detail & Related papers  (2025-05-29T17:59:38Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models.<n>Our framework incorporates two complementary strategies: internal TTC and external TTC.<n>We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAI [58.29510889419971]
  Existing benchmarks for evaluating the security risks and capabilities of code-generating large language models (LLMs) face several key limitations.<n>We introduce a general and scalable benchmark construction framework that begins with manually validated, high-quality seed examples and expands them via targeted mutations.<n>Applying this framework to Python, C/C++, and Java, we build SeCodePLT, a dataset of more than 5.9k samples spanning 44 CWE-based risk categories and three security capabilities.
  arXiv  Detail & Related papers  (2024-10-14T21:17:22Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.