TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications

• URL: http://arxiv.org/abs/2604.17778v1
• Date: Mon, 20 Apr 2026 04:00:13 GMT
• Title: TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications
• Abstract summary: Large language models (LLMs) are increasingly deployed in the telecommunications domain for critical tasks.<n>We introduce TeleEmbedBench, the first large-scale, multi-corpus embedding benchmark designed specifically for telecommunications.
• Score: 2.2508462342902633
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly deployed in the telecommunications domain for critical tasks, relying heavily on Retrieval-Augmented Generation (RAG) to adapt general-purpose models to continuously evolving standards. However, a significant gap exists in evaluating the embedding models that power these RAG pipelines, as general-purpose benchmarks fail to capture the dense, acronym-heavy, and highly cross-referential nature of telecommunications corpora. To address this, we introduce TeleEmbedBench, the first large-scale, multi-corpus embedding benchmark designed specifically for telecommunications. The benchmark spans three heterogeneous corpora: O-RAN Alliance specifications, 3GPP release documents, and the srsRAN open-source codebase, comprising 9,000 question-chunk pairs across three standard chunk sizes (512, 1024, and 2048 tokens). To construct this dataset at scale without manual annotation bottlenecks, we employ a novel automated pipeline where one LLM generates specific queries from text chunks and a secondary LLM validates them across strict criteria. We comprehensively evaluate eight embedding models, spanning standard sentence-transformers and LLM-based embedders. Our results demonstrate that LLM-based embedders, such as Qwen3 and EmbeddingGemma, consistently and significantly outperform traditional sentence-transformers in both retrieval accuracy and robustness against cross-domain interference. Additionally, we introduce TeleEmbedBench-Clean to evaluate model robustness against noisy, incomplete user queries. Finally, our analysis reveals that while domain-specific task instructions improve embedder performance for raw source code, they paradoxically degrade retrieval performance for natural language telecommunications specifications.

Related papers

• TeleSWEBench: A Commit-Driven Benchmark for Evaluating LLM-Powered Software Engineering in Telecommunications [8.423549171787396]
  General-purpose coding benchmarks fail to capture the stateful logic and strict requirements of telecommunications.<n>We introduce TeleSWEBench, the first commit-driven benchmark specifically designed to measure an agent's performance in the telecom domain.
  arXiv  Detail & Related papers  (2026-06-03T15:19:40Z)
• MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks [68.78064578688809]
  MTR-Suite is a unified framework for auditing, synthesizing, and benchmarking retrieval.<n>It features: (1) MTR-Eval, an LLM-based auditor quantifying alignment gaps in previous benchmarks; (2) MTR-Pipeline, a multi-agent system using greedy clustering to generate high-fidelity dialogues at 1/400th human cost; and (3) MTR-Bench, a rigorous general-domain benchmark.
  arXiv  Detail & Related papers  (2026-05-20T05:26:35Z)
• TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications? [12.347241408492279]
  We present TeleCom-Bench, a comprehensive benchmark comprising 12 evaluation sets with 22,678 curated samples, which evaluates Large Language Models.<n>We show that current LLMs function competently as diagnosticians but fail as field engineers.<n>This capability gap demonstrates that current LLMs function competently as diagnosticians but fail as field engineers.
  arXiv  Detail & Related papers  (2026-05-18T08:14:49Z)
• Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions [6.606052122056915]
  textscIntent2Tx grounds natural language intents in real-world protocol interactions across 11 categories.<n>textscIntent2Tx serves as a critical foundation for developing autonomous, reliable agents in intent-centric Web3 ecosystems.
  arXiv  Detail & Related papers  (2026-04-30T11:52:50Z)
• Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions [0.0]
  Existing benchmarks evaluate a single detector on a single dataset under ideal conditions.<n>We present a benchmark evaluating diverse detection approaches across two corpora.<n>Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift.
  arXiv  Detail & Related papers  (2026-03-18T09:27:27Z)
• RAGTrack: Language-aware RGBT Tracking with Retrieval-Augmented Generation [71.2136732268131]
  RGB-Thermal (RGBT) tracking aims to achieve robust object localization across diverse environmental conditions.<n>Existing RGBT trackers rely solely on initial-frame visual information for target modeling.<n>We propose RAGTrack, a novel Retrieval-Augmented Generation framework for robust RGBT tracking.
  arXiv  Detail & Related papers  (2026-03-04T01:02:04Z)
• RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis [78.32151470154422]
  We introduce RAVEL, an agentic framework that enables the testers to autonomously plan and execute typical synthesis operations.<n>We present C3EBench, a benchmark comprising 1,258 samples derived from professional human writings.<n>By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability.
  arXiv  Detail & Related papers  (2026-02-28T14:47:34Z)
• A Generalizable Framework for Building Executable Domain-Specific LLMs under Data Scarcity: Demonstration on Semiconductor TCAD Simulation [20.174394305112198]
  We present a framework for building compact, executable domain-specific LLMs in low-resource settings.<n>We demonstrate the framework by instantiating TcadGPT for semiconductor Technology Computer-Aided Design (TCAD)<n>Using 1.5M synthetic QA pairs and an IR-driven DPO dataset, TcadGPT attains 85.6% semantic accuracy and an 80.0% syntax pass rate on SDE executability tests.
  arXiv  Detail & Related papers  (2026-01-15T07:13:34Z)
• How to Discover Knowledge for FutureG: Contextual RAG and LLM Prompting for O-RAN [5.589132273787205]
  We present a retrieval-augmented question answering framework for 5G/6G networks, where the Open Radio Access Network (O-RAN) has become central to disaggregated, AI-driven wireless systems.<n>To address this challenge, we adopt Contextual Retrieval-Augmented Generation (Contextual RAG)<n>We show that Contextual RAG consistently improves accuracy over standard RAG and runtime, while maintaining competitive runtime and CO2 emissions.
  arXiv  Detail & Related papers  (2025-12-18T18:03:59Z)
• RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering [50.42577862494645]
  We present RAG-IGBench, a benchmark designed to evaluate the task of Interleaved Generation based on Retrieval-Augmented Generation (RAG-IG) in open-domain question answering.<n>RAG-IG integrates multimodal large language models (MLLMs) with retrieval mechanisms, enabling the models to access external image-text information for generating coherent multimodal content.
  arXiv  Detail & Related papers  (2025-10-11T03:06:39Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Large Language Models are Good Relational Learners [55.40941576497973]
  We introduce Rel-LLM, a novel architecture that utilizes a graph neural network (GNN)- based encoder to generate structured relational prompts for large language models (LLMs)<n>Unlike traditional text-based serialization approaches, our method preserves the inherent relational structure of databases while enabling LLMs to process and reason over complex entity relationships.
  arXiv  Detail & Related papers  (2025-06-06T04:07:55Z)
• TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models [7.015008083968722]
  Large Language Models (LLMs) have the potential to revolutionize the Sixth Generation (6G) communication networks. This paper proposes a pipeline to adapt any general purpose LLMs to a telecom-specific LLMs. We extend existing evaluation benchmarks and proposed three new benchmarks, namely, Telecom Math Modeling, Telecom Open QnA and Telecom Code Tasks.
  arXiv  Detail & Related papers  (2024-07-12T16:51:02Z)
• Rethinking Text Line Recognition Models [57.47147190119394]
  We consider two decoder families (Connectionist Temporal Classification and Transformer) and three encoder modules (Bidirectional LSTMs, Self-Attention, and GRCLs) We compare their accuracy and performance on widely used public datasets of scene and handwritten text. Unlike the more common Transformer-based models, this architecture can handle inputs of arbitrary length.
  arXiv  Detail & Related papers  (2021-04-15T21:43:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.