Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewriting

• URL: http://arxiv.org/abs/2404.12283v1
• Date: Thu, 18 Apr 2024 15:58:56 GMT
• Title: Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewriting
• Authors: Nicholas Harris, Anand Butani, Syed Hashmy,
• Abstract summary: This paper proposes a novel approach to improve embedding performance by leveraging large language models (LLMs) to enrich and rewrite input text before the embedding process. The effectiveness of this approach is evaluated on three datasets: Banking77Classification, TwitterSemEval 2015, and Amazon Counter-factual Classification.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embedding models are crucial for various natural language processing tasks but can be limited by factors such as limited vocabulary, lack of context, and grammatical errors. This paper proposes a novel approach to improve embedding performance by leveraging large language models (LLMs) to enrich and rewrite input text before the embedding process. By utilizing ChatGPT 3.5 to provide additional context, correct inaccuracies, and incorporate metadata, the proposed method aims to enhance the utility and accuracy of embedding models. The effectiveness of this approach is evaluated on three datasets: Banking77Classification, TwitterSemEval 2015, and Amazon Counter-factual Classification. Results demonstrate significant improvements over the baseline model on the TwitterSemEval 2015 dataset, with the best-performing prompt achieving a score of 85.34 compared to the previous best of 81.52 on the Massive Text Embedding Benchmark (MTEB) Leaderboard. However, performance on the other two datasets was less impressive, highlighting the importance of considering domain-specific characteristics. The findings suggest that LLM-based text enrichment has shown promising results to improve embedding performance, particularly in certain domains. Hence, numerous limitations in the process of embedding can be avoided.

Related papers

• Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback [50.84142264245052]
  This work introduces the Align-SLM framework to enhance the semantic understanding of textless Spoken Language Models (SLMs) Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO) We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation.
  arXiv  Detail & Related papers  (2024-11-04T06:07:53Z)
• Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores [12.86467344792873]
  The proposed method uses soft labels derived from expert-augmented scores to fine-tune embedding models. The paper evaluates the method using a Q&A dataset from an online shopping website and eight expert models.
  arXiv  Detail & Related papers  (2024-08-19T01:59:25Z)
• Improving Text Embeddings for Smaller Language Models Using Contrastive Fine-tuning [0.9561495813823734]
  We conduct contrastive fine-tuning on the NLI dataset. MiniCPM shows the most significant improvements of an average 56.33% performance gain.
  arXiv  Detail & Related papers  (2024-08-01T16:31:35Z)
• Language Models for Text Classification: Is In-Context Learning Enough? [54.869097980761595]
  Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches is the ability to understand instructions written in natural language (prompts) This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances.
  arXiv  Detail & Related papers  (2024-03-26T12:47:39Z)
• Improving Sampling Methods for Fine-tuning SentenceBERT in Text Streams [49.3179290313959]
  This study explores the efficacy of seven text sampling methods designed to selectively fine-tune language models. We precisely assess the impact of these methods on fine-tuning the SBERT model using four different loss functions. Our findings indicate that Softmax loss and Batch All Triplets loss are particularly effective for text stream classification.
  arXiv  Detail & Related papers  (2024-03-18T23:41:52Z)
• Repetition Improves Language Model Embeddings [68.92976440181387]
  We propose "echo embeddings," in which we repeat the input twice in context and extract embeddings from the second occurrence. On the MTEB leaderboard, echo embeddings improve over classical embeddings by over 9% zero-shot and by around 0.7% when fine-tuned.
  arXiv  Detail & Related papers  (2024-02-23T17:25:10Z)
• How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
  The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges. We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
  arXiv  Detail & Related papers  (2023-11-14T08:51:00Z)
• Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models [4.451741472324815]
  Jina Embeddings constitutes a set of high-performance sentence embedding models adept at translating textual inputs into numerical representations. This paper details the development of Jina Embeddings, starting with the creation of high-quality pairwise and triplet datasets. It concludes with a comprehensive performance evaluation using the Massive Text Embedding Benchmark (MTEB)
  arXiv  Detail & Related papers  (2023-07-20T20:37:24Z)
• ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models [102.63817106363597]
  We build ELEVATER, the first benchmark to compare and evaluate pre-trained language-augmented visual models. It consists of 20 image classification datasets and 35 object detection datasets, each of which is augmented with external knowledge. We will release our toolkit and evaluation platforms for the research community.
  arXiv  Detail & Related papers  (2022-04-19T10:23:42Z)
• Explaining and Improving BERT Performance on Lexical Semantic Change Detection [22.934650688233734]
  Recent success of type-based models in SemEval-2020 Task 1 has raised the question why the success of token-based models does not translate to our field. We investigate the influence of a range of variables on clusterings of BERT vectors and show that its low performance is largely due to orthographic information on the target word.
  arXiv  Detail & Related papers  (2021-03-12T13:29:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.