TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?

• URL: http://arxiv.org/abs/2605.18025v1
• Date: Mon, 18 May 2026 08:14:49 GMT
• Title: TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?
• Abstract summary: We present TeleCom-Bench, a comprehensive benchmark comprising 12 evaluation sets with 22,678 curated samples, which evaluates Large Language Models.<n>We show that current LLMs function competently as diagnosticians but fail as field engineers.<n>This capability gap demonstrates that current LLMs function competently as diagnosticians but fail as field engineers.
• Score: 12.347241408492279
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Language Models have achieved remarkable integration in various vertical scenarios, their deployment in the telecommunications domain remains exploratory due to the lack of a standardized evaluation framework. Current telecom benchmarks primarily focus on static, foundational knowledge and isolated atomic skills, neglecting the equipment-specific documentation and end-to-end industrial workflows essential for real-world production systems. To bridge this gap, we present TeleCom-Bench, a comprehensive benchmark comprising 12 evaluation sets with 22,678 curated samples, which evaluates LLMs across a synergistic hierarchy: (1) Multi-dimensional Knowledge Comprehension, which integrates telecommunication fundamentals, 3GPP protocols, and 5G network architecture with proprietary product knowledge across wired, core, and wireless networks via knowledge graph-driven synthesis; and (2)End-to-End Knowledge Application, which formalizes six core tasks on authentic trajectories from live network agent workflows, including intent recognition, entity extraction, event verification, tool invocation, root cause analysis, and solution generation-across network optimization and fault maintenance scenarios. Evaluations of eight state-of-the-art LLMs reveal a universal Execution Wall: while models achieve 90% accuracy in linguistic interface tasks such as intent recognition and entity extraction, performance collapses to approximately 30% in procedural execution tasks like solution generation. This capability gap demonstrates that current LLMs function competently as diagnosticians but fail as field engineers. TeleCom-Bench provides standardized diagnostics to precisely pinpoint this deficit, offering actionable guidance for domain-specific alignment toward production-ready telecom agents. The dataset and evaluation code have been released at https://github.com/ZTE-AICloud/TeleCom-Bench.

Related papers

• OpenCompass: A Universal Evaluation Platform for Large Language Models [62.59671563145442]
  General-purpose large language models (LLMs) have become a critical link in advancing technological development.<n>Mainstream benchmark datasets face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing.<n>This paper proposes and open-source Open LLMs, a one-stop, scalable, and high-concurrency general-purpose evaluation platform.
  arXiv  Detail & Related papers  (2026-05-19T02:50:11Z)
• Towards Agentic Test-Driven Quality Assurance for 6G Networks [0.0]
  This work proposes an agentic, intent-driven end-to-end (E2E) orchestration framework that integrates intent co-creation with a Test-Driven Quality Assurance paradigm.<n>The architecture is grounded in a standards-aligned knowledge representation using TM Forum information models and catalogs.
  arXiv  Detail & Related papers  (2026-04-25T12:58:36Z)
• TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications [2.2508462342902633]
  Large language models (LLMs) are increasingly deployed in the telecommunications domain for critical tasks.<n>We introduce TeleEmbedBench, the first large-scale, multi-corpus embedding benchmark designed specifically for telecommunications.
  arXiv  Detail & Related papers  (2026-04-20T04:00:13Z)
• CrossTraffic: An Open-Source Framework for Reproducible and Executable Transportation Analysis and Knowledge Management [18.821780833520116]
  CrossTraffic is an open-source framework that treats transportation methodologies and regulatory knowledge as continuously deployable and verifiable software infrastructure.<n>CrossTraffic provides an executable computational core for transportation analysis with cross-platform access through standardized interfaces.
  arXiv  Detail & Related papers  (2026-02-08T22:30:27Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• ThinkGeo: Evaluating Tool-Augmented Agents for Remote Sensing Tasks [64.86209459039313]
  ThinkGeo is an agentic benchmark designed to evaluate tool-augmented agents on remote sensing tasks via structured tool use and multi-step planning.<n>We implement a ReAct-style interaction loop and evaluate both open and closed-source LLMs on 486 structured agentic tasks with 1,773 expert-verified reasoning steps.<n>Our analysis reveals notable disparities in tool accuracy and planning consistency across models.
  arXiv  Detail & Related papers  (2025-05-29T17:59:38Z)
• Understanding 6G through Language Models: A Case Study on LLM-aided Structured Entity Extraction in Telecom Domain [55.627646392044824]
  This work proposes a novel language model-based information extraction technique, aiming to extract structured entities from the telecom context.<n>The proposed telecom structured entity extraction (TeleSEE) technique applies a token-efficient representation method to predict entity types and attribute keys, aiming to save the number of output tokens and improve prediction accuracy.
  arXiv  Detail & Related papers  (2025-05-20T21:00:08Z)
• Efficient Telecom Specific LLM: TSLAM-Mini with QLoRA and Digital Twin Data [0.0]
  General-purpose large language models (LLMs) frequently exhibit suboptimal performance in real-time telecommunications applications.<n>This investigation addresses this critical limitation through the meticulous fine-tuning of TSLAM-Mini developed by NetoAI.
  arXiv  Detail & Related papers  (2025-05-10T12:28:47Z)
• TeleEval-OS: Performance evaluations of large language models for operations scheduling [34.77222716408485]
  We propose the first Telecommunications Operation Scheduling Evaluation Benchmark (TeleEval-OS)<n>This benchmark comprises 15 datasets across 13 subtasks, comprehensively simulating four key operational stages: intelligent ticket creation, intelligent ticket handling, intelligent ticket closure, and intelligent evaluation.<n>We categorize their capabilities in telecommunications operation scheduling into four hierarchical levels, arranged in ascending order of difficulty: basic NLP, knowledge Q&A, report generation, and report analysis.
  arXiv  Detail & Related papers  (2025-05-06T02:44:41Z)
• Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generation [52.8352968531863]
  Large language models (LLMs) have made significant progress in general-purpose natural language processing tasks.<n>This paper presents a novel framework that combines knowledge graph (KG) and retrieval-augmented generation (RAG) techniques to enhance LLM performance in the telecom domain.
  arXiv  Detail & Related papers  (2025-03-31T15:58:08Z)
• Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G Networks [84.2155885234293]
  We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC. To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
  arXiv  Detail & Related papers  (2020-02-22T14:38:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.