TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents

• URL: http://arxiv.org/abs/2604.06209v1
• Date: Mon, 16 Mar 2026 07:02:30 GMT
• Title: TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents
• Abstract summary: We introduce TelcoAgent-Bench and TelcoAgent-Metrics, a Telecom-specific benchmarking framework for evaluating multilingual telecom LLM agents.<n>Our contribution includes a structured suite of metrics that assess intent recognition, ordered tool execution, resolution correctness, and stability across scenario variations.<n>The framework is designed to operate in both English and Arabic, to address the need for multilingual agent deployment in operational network environments.
• Score: 1.7172745214009204
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The integration of large language model (LLM) agents into telecom networks introduces new challenges, related to intent recognition, tool execution, and resolution generation, while taking into consideration different operational constraints. In this paper, we introduce TelcoAgent-Bench and TelcoAgent-Metrics, a Telecom-specific benchmarking framework for evaluating multilingual telecom LLM agents. The proposed framework assesses the semantic understanding as well as process-level alignment with structured troubleshooting flows and stability across repeated scenario variations. Our contribution includes a structured suite of metrics that assess intent recognition, ordered tool execution, resolution correctness, and stability across scenario variations, with the aim of quantifying the reliability and operational consistency of LLM agents in telecom environments. The framework is designed to operate in both English and Arabic, to address the need for multilingual agent deployment in operational network environments. Our experimental results show that although recent instruct-tuned models can understand telecom problems in a reasonable way, they usually struggle to consistently follow the required troubleshooting steps and to maintain stable behavior when exposed to different variations of the same scenario. This performance gap becomes more pronounced in unconstrained and bilingual settings.

Related papers

• Diffusion Language Models for Mobile Edge Agentic AI: Foundations, Applications, and Challenges [58.887152635127165]
  Diffusion language models (DLMs) offer a non-autoregressive alternative for mobile edge agentic artificial intelligence (AI)<n>DLMs can update multiple uncertain tokens in parallel and exploit bidirectional context throughout the generation process.<n>This survey reviews DLM foundations and analyzes their suitability for edge settings under latency, memory, energy, bandwidth, privacy, and reliability constraints.
  arXiv  Detail & Related papers  (2026-09-04T06:16:49Z)
• PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents [72.96251271825038]
  Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments.<n>Most existing benchmarks implicitly assume a monolingual setting, where the entire execution process is conducted within a single language.<n>We introduce PolyWorkBench, a benchmark for evaluating agents on multilingual long-horizon workplace work.
  arXiv  Detail & Related papers  (2026-07-07T08:50:09Z)
• FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments [60.3427704389541]
  Large Language Models are being increasingly deployed as the decision-making core of autonomous agents.<n>Yet, in conversational benchmarks, these agents frequently fail due to the cascading effects of incorrect decision-making.<n>We present the Failure-Aware Meta-Agentic framework to tackle these challenges.
  arXiv  Detail & Related papers  (2026-04-28T02:21:53Z)
• Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agents [35.76774274440008]
  STING (Sequential Testing of Illicit N-step Goal execution) is an automated red-teaming framework.<n>It constructs a step-by-step illicit plan grounded in a benign persona and iteratively probes a target agent with adaptive follow-ups.<n>We introduce an analysis framework that models multi-turn red-teaming as a time-to-first-jailbreak random variable.
  arXiv  Detail & Related papers  (2026-02-18T10:31:19Z)
• ComAgent: Multi-LLM based Agentic AI Empowered Intelligent Wireless Networks [62.031889234230725]
  6G networks rely on complex cross-layer optimization.<n> manually translating high-level intents into mathematical formulations remains a bottleneck.<n>We present ComAgent, a multi-LLM agentic AI framework.
  arXiv  Detail & Related papers  (2026-01-27T13:43:59Z)
• Bridging the Knowledge Void: Inference-time Acquisition of Unfamiliar Programming Languages for Coding Tasks [22.908904483320953]
  Large Language Models (LLMs) in coding tasks are often a reflection of their extensive pre-training corpora.<n>We propose ILA-agent, a general ILA framework that equips LLMs with a set of behavioral primitives.<n>We instantiate ILA-agent for Cangjie and evaluate its performance across code generation, translation, and program repair tasks.
  arXiv  Detail & Related papers  (2026-01-16T09:06:47Z)
• MM-Telco: Benchmarks and Multimodal Large Language Models for Telecom Applications [9.282400615034534]
  Large Language Models (LLMs) have emerged as powerful tools for automating complex reasoning and decision-making tasks.<n>In telecommunications, they hold the potential to transform network optimization, automate troubleshooting, enhance customer support, and ensure regulatory compliance.<n>We propose MM-Telco, a comprehensive suite of benchmarks and models tailored for the telecom domain.
  arXiv  Detail & Related papers  (2025-11-17T08:34:41Z)
• SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models [59.90381306452982]
  evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer.<n>We introduce SWE-1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework.<n>SWE- spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests.
  arXiv  Detail & Related papers  (2025-11-07T18:01:32Z)
• ContextNav: Towards Agentic Multimodal In-Context Learning [85.05420047017513]
  ContextNav is an agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation.<n>It builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts.<n> Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets.
  arXiv  Detail & Related papers  (2025-10-06T07:49:52Z)
• cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending [3.939989712024268]
  Contextual MARL (cMARL) addresses this by parameterizing environments with context variables and training a context-agnostic policy.<n>Existing cMARL methods attempt to use curriculum learning to help train and evaluate context-agnostic policies.<n>We propose Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blending (cMALC-D)
  arXiv  Detail & Related papers  (2025-08-28T14:16:17Z)
• A Pilot Study on LLM-Based Agentic Translation from Android to iOS: Pitfalls and Insights [27.632954643254884]
  Large Language Models (LLMs) were recently leveraged to enhance code translation at different granularities.<n>This study aims to fill this gap by evaluating LLM-based agentic approaches for mobile application translation.<n>We developed a chain of agents that account for dependencies, specifications, program structure, and program control flow.
  arXiv  Detail & Related papers  (2025-07-21T20:11:01Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.