Context-Aware Abbreviation Expansion Using Large Language Models

• URL: http://arxiv.org/abs/2205.03767v3
• Date: Wed, 11 May 2022 02:25:35 GMT
• Title: Context-Aware Abbreviation Expansion Using Large Language Models
• Authors: Shanqing Cai, Subhashini Venugopalan, Katrin Tomanek, Ajit Narayanan, Meredith Ringel Morris, Michael P. Brenner
• Abstract summary: We propose a paradigm in which phrases are abbreviated aggressively as primarily word-initial letters. Our approach is to expand the abbreviations into full-phrase options by leveraging conversation context.
• Score: 16.52516727224014
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Motivated by the need for accelerating text entry in augmentative and alternative communication (AAC) for people with severe motor impairments, we propose a paradigm in which phrases are abbreviated aggressively as primarily word-initial letters. Our approach is to expand the abbreviations into full-phrase options by leveraging conversation context with the power of pretrained large language models (LLMs). Through zero-shot, few-shot, and fine-tuning experiments on four public conversation datasets, we show that for replies to the initial turn of a dialog, an LLM with 64B parameters is able to exactly expand over 70% of phrases with abbreviation length up to 10, leading to an effective keystroke saving rate of up to about 77% on these exact expansions. Including a small amount of context in the form of a single conversation turn more than doubles abbreviation expansion accuracies compared to having no context, an effect that is more pronounced for longer phrases. Additionally, the robustness of models against typo noise can be enhanced through fine-tuning on noisy data.

Related papers

• Retrieval Augmented End-to-End Spoken Dialog Models [20.896330994089283]
  We apply SLM to speech dialog applications where the dialog states are inferred directly from the audio signal. Inspired by the RAG (retrieval-augmented generation) paradigm, we propose a retrieval augmented SLM (ReSLM) that overcomes this weakness. We evaluated ReSLM on speech MultiWoz task (DSTC-11 challenge), and found that this retrieval augmentation boosts model performance.
  arXiv  Detail & Related papers  (2024-02-02T18:23:09Z)
• Parameter Efficient Tuning Allows Scalable Personalization of LLMs for Text Entry: A Case Study on Abbreviation Expansion [14.366537646319946]
  Abbreviation expansion is a strategy used to speed up communication by limiting the amount of typing and using a language model to suggest expansions. Here we look at personalizing a Large Language Model's (LLM) suggestions based on prior conversations to enhance the relevance of predictions. We compare fine-tuning, prompt-tuning, and retrieval augmented generation of expanded text suggestions for abbreviated inputs.
  arXiv  Detail & Related papers  (2023-12-21T22:52:44Z)
• Locally Differentially Private Document Generation Using Zero Shot Prompting [61.20953109732442]
  We propose a locally differentially private mechanism called DP-Prompt to counter author de-anonymization attacks. When DP-Prompt is used with a powerful language model like ChatGPT (gpt-3.5), we observe a notable reduction in the success rate of de-anonymization attacks.
  arXiv  Detail & Related papers  (2023-10-24T18:25:13Z)
• Enhancing Argument Structure Extraction with Efficient Leverage of Contextual Information [79.06082391992545]
  We propose an Efficient Context-aware model (ECASE) that fully exploits contextual information. We introduce a sequence-attention module and distance-weighted similarity loss to aggregate contextual information and argumentative information. Our experiments on five datasets from various domains demonstrate that our model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-10-08T08:47:10Z)
• Keyword Augmented Retrieval: Novel framework for Information Retrieval integrated with speech interface [0.0]
  Retrieving answers in a quick and low cost manner without hallucinations using Language models is a major hurdle. This is what prevents employment of Language models in knowledge retrieval automation. For commercial search and chat-bot applications, complete reliance on commercial large language models (LLMs) like GPT 3.5 etc. can be very costly.
  arXiv  Detail & Related papers  (2023-10-06T12:44:04Z)
• ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation [59.91139600152296]
  ParaAMR is a large-scale syntactically diverse paraphrase dataset created by abstract meaning representation back-translation. We show that ParaAMR can be used to improve on three NLP tasks: learning sentence embeddings, syntactically controlled paraphrase generation, and data augmentation for few-shot learning.
  arXiv  Detail & Related papers  (2023-05-26T02:27:33Z)
• SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents [72.42049370297849]
  SpokenWOZ is a large-scale speech-text dataset for spoken TOD. Cross-turn slot and reasoning slot detection are new challenges for SpokenWOZ.
  arXiv  Detail & Related papers  (2023-05-22T13:47:51Z)
• Short-Term Word-Learning in a Dynamically Changing Environment [63.025297637716534]
  We show how to supplement an end-to-end ASR system with a word/phrase memory and a mechanism to access this memory to recognize the words and phrases correctly. We demonstrate significant improvements in the detection rate of new words with only a minor increase in false alarms.
  arXiv  Detail & Related papers  (2022-03-29T10:05:39Z)
• Structured abbreviation expansion in context [12.000998471674649]
  We consider the task of reversing ad hoc abbreviations in context to recover normalized, expanded versions of abbreviated messages. The problem is related to, but distinct from, spelling correction, in that ad hoc abbreviations are intentional and may involve substantial differences from the original words.
  arXiv  Detail & Related papers  (2021-10-04T01:22:43Z)
• Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration [25.159601117722936]
  We propose a contrastive fine-tuning objective that enables BERT to produce more powerful phrase embeddings. Our approach relies on a dataset of diverse phrasal paraphrases, which is automatically generated using a paraphrase generation model. As a case study, we show that Phrase-BERT embeddings can be easily integrated with a simple autoencoder to build a phrase-based neural topic model.
  arXiv  Detail & Related papers  (2021-09-13T20:31:57Z)
• Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification [68.3291372168167]
  We focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT) We expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.
  arXiv  Detail & Related papers  (2021-08-04T13:00:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.