QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling

• URL: http://arxiv.org/abs/2409.14175v2
• Date: Tue, 04 Feb 2025 07:46:23 GMT
• Title: QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling
• Authors: Blessed Guda, Gabrial Zencha Ashungafac, Lawrence Francis, Carlee Joe-Wong,
• Abstract summary: GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation framework. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain.
• Score: 10.42541749928513
• License:
• Abstract: Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.

Related papers

• Empowering Large Language Models in Wireless Communication: A Novel Dataset and Fine-Tuning Framework [81.29965270493238]
  We develop a specialized dataset aimed at enhancing the evaluation and fine-tuning of large language models (LLMs) for wireless communication applications. The dataset includes a diverse set of multi-hop questions, including true/false and multiple-choice types, spanning varying difficulty levels from easy to hard. We introduce a Pointwise V-Information (PVI) based fine-tuning method, providing a detailed theoretical analysis and justification for its use in quantifying the information content of training data.
  arXiv  Detail & Related papers  (2025-01-16T16:19:53Z)
• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities. LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands. We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents [27.112239616508834]
  Mixture of Refinement Agents (MoRA) is a novel agentic refinement framework for large language models (LLMs) MoRA iteratively refines the LLM generated base solution by correcting the aforementioned errors, resulting in a significant performance improvement for open-source LLMs. We evaluate our approach on the SciEval and MMLU subsets along with our own physics dataset (PhysicsQA)
  arXiv  Detail & Related papers  (2024-12-01T14:15:55Z)
• DRS: Deep Question Reformulation With Structured Output [114.14122339938697]
  Large language models (LLMs) can detect unanswerable questions, but struggle to assist users in reformulating these questions. We propose DRS: Deep Question Reformulation with Structured Output, a novel zero-shot method aimed at enhancing LLMs ability to assist users in reformulating questions. We show that DRS improves the reformulation accuracy of GPT-3.5 from 23.03% to 70.42%, while also enhancing the performance of open-source models, such as Gemma2-9B, from 26.35% to 56.75%.
  arXiv  Detail & Related papers  (2024-11-27T02:20:44Z)
• mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA [78.45521005703958]
  multimodal Retrieval-Augmented Generation (mRAG) is naturally introduced to provide MLLMs with comprehensive and up-to-date knowledge. We propose a novel framework called textbfRetrieval-textbfReftextbfAugmented textbfGeneration (mR$2$AG) which achieves adaptive retrieval and useful information localization. mR$2$AG significantly outperforms state-of-the-art MLLMs on INFOSEEK and Encyclopedic-VQA
  arXiv  Detail & Related papers  (2024-11-22T16:15:50Z)
• TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network [4.551436852242372]
  We present TeleOracle, a telecom-specialized retrieval-augmented generation (RAG) system built on the Phi-2 small language model (SLM) To improve context retrieval, TeleOracle employs a two-stage retriever that incorporates semantic chunking and hybrid keyword and semantic search. A thorough analysis of the model's performance indicates that our RAG framework is effective in aligning Phi-2 to the telecom domain in a downstream question and answer (QnA) task, achieving a 30% improvement in accuracy over the base Phi-2 model.
  arXiv  Detail & Related papers  (2024-11-04T21:12:08Z)
• GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input. GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak)
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• Rephrase and Contrast: Fine-Tuning Language Models for Enhanced Understanding of Communication and Computer Networks [13.829525575305206]
  This paper introduces our Rephrase and Contrast (RaC) framework, an efficient fine-tuning framework. RaC enhances LLMs' comprehension and critical thinking abilities by incorporating question reformulation and contrastive analysis. To efficiently construct the dataset for RaC fine-tuning, we develop a GPT-assisted data mining method for generating high-quality question-answer pairs.
  arXiv  Detail & Related papers  (2024-09-21T16:04:43Z)
• Large Language Model Enhanced Multi-Agent Systems for 6G Communications [94.45712802626794]
  We propose a multi-agent system with customized communication knowledge and tools for solving communication related tasks using natural language. We validate the effectiveness of the proposed multi-agent system by designing a semantic communication system.
  arXiv  Detail & Related papers  (2023-12-13T02:35:57Z)
• Analysis of the Reasoning with Redundant Information Provided Ability of Large Language Models [0.0]
  Large Language Models (LLMs) have demonstrated impressive capabilities across a range of natural language processing tasks. To address this gap, a new form of Question-Answering (QA) task, termed Reasoning with Redundant Information Provided (RRIP), is introduced. This study evaluates two popular LLMs, LlaMA2-13B-chat and generative pre-trained transformer 3.5 (GPT-3.5), contrasting their performance on traditional QA tasks against RRIP tasks.
  arXiv  Detail & Related papers  (2023-10-06T06:20:06Z)
• Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities [32.035405009895264]
  Large language models (LLMs) are revolutionizing AI development and potentially shaping our future. The status quo cloud-based deployment faces some critical challenges: 1) long response time; 2) high bandwidth costs; and 3) the violation of data privacy. 6G mobile edge computing (MEC) systems may resolve these pressing issues. This article serves as a position paper for thoroughly identifying the motivation, challenges, and pathway for empowering LLMs at the 6G edge.
  arXiv  Detail & Related papers  (2023-09-28T06:22:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.