TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models

• URL: http://arxiv.org/abs/2407.09424v1
• Date: Fri, 12 Jul 2024 16:51:02 GMT
• Title: TelecomGPT: A Framework to Build Telecom-Specfic Large Language Models
• Authors: Hang Zou, Qiyang Zhao, Yu Tian, Lina Bariah, Faouzi Bader, Thierry Lestable, Merouane Debbah,
• Abstract summary: Large Language Models (LLMs) have the potential to revolutionize the Sixth Generation (6G) communication networks. This paper proposes a pipeline to adapt any general purpose LLMs to a telecom-specific LLMs. We extend existing evaluation benchmarks and proposed three new benchmarks, namely, Telecom Math Modeling, Telecom Open QnA and Telecom Code Tasks.
• Score: 7.015008083968722
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models (LLMs) have the potential to revolutionize the Sixth Generation (6G) communication networks. However, current mainstream LLMs generally lack the specialized knowledge in telecom domain. In this paper, for the first time, we propose a pipeline to adapt any general purpose LLMs to a telecom-specific LLMs. We collect and build telecom-specific pre-train dataset, instruction dataset, preference dataset to perform continual pre-training, instruct tuning and alignment tuning respectively. Besides, due to the lack of widely accepted evaluation benchmarks in telecom domain, we extend existing evaluation benchmarks and proposed three new benchmarks, namely, Telecom Math Modeling, Telecom Open QnA and Telecom Code Tasks. These new benchmarks provide a holistic evaluation of the capabilities of LLMs including math modeling, Open-Ended question answering, code generation, infilling, summarization and analysis in telecom domain. Our fine-tuned LLM TelecomGPT outperforms state of the art (SOTA) LLMs including GPT-4, Llama-3 and Mistral in Telecom Math Modeling benchmark significantly and achieve comparable performance in various evaluation benchmarks such as TeleQnA, 3GPP technical documents classification, telecom code summary and generation and infilling.

Related papers

• Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications [20.36003316123051]
  We develop and open-source Tele-LLMs, the first series of language models ranging from 1B to 8B parameters, specifically tailored for telecommunications. Our evaluations demonstrate that these models outperform their general-purpose counterparts on Tele-Eval while retaining their previously acquired capabilities.
  arXiv  Detail & Related papers  (2024-09-09T03:58:51Z)
• TelecomRAG: Taming Telecom Standards with Retrieval Augmented Generation and LLMs [7.67846565247214]
  Large Language Models (LLMs) have immense potential to transform the telecommunications industry. LLMs could help professionals understand complex standards, generate code, and accelerate development. Retrieval-augmented generation (RAG) offers a way to create precise, fact-based answers.
  arXiv  Detail & Related papers  (2024-06-11T08:35:23Z)
• MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series [86.31735321970481]
  We open-source MAP-Neo, a bilingual language model with 7B parameters trained from scratch on 4.5T high-quality tokens. Our MAP-Neo is the first fully open-sourced bilingual LLM with comparable performance compared to existing state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2024-05-29T17:57:16Z)
• Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities [36.711166825551715]
  Large language models (LLMs) have received considerable attention recently due to their outstanding comprehension and reasoning capabilities. This work aims to provide a comprehensive overview of LLM-enabled telecom networks.
  arXiv  Detail & Related papers  (2024-05-17T14:46:13Z)
• WDMoE: Wireless Distributed Large Language Models with Mixture of Experts [65.57581050707738]
  We propose a wireless distributed Large Language Models (LLMs) paradigm based on Mixture of Experts (MoE) We decompose the MoE layer in LLMs by deploying the gating network and the preceding neural network layer at base station (BS) and mobile devices. We design an expert selection policy by taking into account both the performance of the model and the end-to-end latency.
  arXiv  Detail & Related papers  (2024-05-06T02:55:50Z)
• Using Large Language Models to Understand Telecom Standards [35.343893798039765]
  Large Language Models (LLMs) may provide faster access to relevant information. We evaluate the capability of state-of-art LLMs to be used as Question Answering (QA) assistants. Results show that LLMs can be used as a credible reference tool on telecom technical documents.
  arXiv  Detail & Related papers  (2024-04-02T09:54:51Z)
• Large Language Models: A Survey [69.72787936480394]
  Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data.
  arXiv  Detail & Related papers  (2024-02-09T05:37:09Z)
• TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data [73.29220562541204]
  We consider harnessing the amazing power of language models (LLMs) to solve our task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets.
  arXiv  Detail & Related papers  (2024-01-24T04:28:50Z)
• Vision-Language Instruction Tuning: A Review and Analysis [52.218690619616474]
  Vision-Language Instruction Tuning (VLIT) presents more complex characteristics compared to pure text instruction tuning. We offer a detailed categorization for existing VLIT datasets and identify the characteristics that high-quality VLIT data should possess. By incorporating these characteristics as guiding principles into the existing VLIT data construction process, we conduct extensive experiments and verify their positive impact on the performance of tuned multi-modal LLMs.
  arXiv  Detail & Related papers  (2023-11-14T14:02:32Z)
• TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge [26.302396162473293]
  TeleQnA is the first benchmark dataset designed to evaluate the knowledge of Large Language Models (LLMs) in telecommunications. This paper outlines the automated question generation framework responsible for creating this dataset, along with how human input was integrated at various stages to ensure the quality of the questions. The dataset has been made publicly accessible on GitHub.
  arXiv  Detail & Related papers  (2023-10-23T15:55:15Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.