AnglE-optimized Text Embeddings

• URL: http://arxiv.org/abs/2309.12871v8
• Date: Wed, 17 Jul 2024 14:33:21 GMT
• Title: AnglE-optimized Text Embeddings
• Authors: Xianming Li, Jing Li,
• Abstract summary: This paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. Extensive experiments were conducted on various tasks including short-text STS, long-text STS, and domain-specific STS tasks.
• Score: 4.545354973721937
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding models face is the problem of vanishing gradients, primarily due to their reliance on the cosine function in the optimization objective, which has saturation zones. To address this issue, this paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. This novel approach effectively mitigates the adverse effects of the saturation zone in the cosine function, which can impede gradient and hinder optimization processes. To set up a comprehensive STS evaluation, we experimented on existing short-text STS datasets and a newly collected long-text STS dataset from GitHub Issues. Furthermore, we examine domain-specific STS scenarios with limited labeled data and explore how AnglE works with LLM-annotated data. Extensive experiments were conducted on various tasks including short-text STS, long-text STS, and domain-specific STS tasks. The results show that AnglE outperforms the state-of-the-art (SOTA) STS models that ignore the cosine saturation zone. These findings demonstrate the ability of AnglE to generate high-quality text embeddings and the usefulness of angle optimization in STS.

Related papers

• TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings [61.9257731511557]
  We propose Text Guided LLaVA (TG-LLaVA) to optimize vision-language models (VLMs) We use learnable latent embeddings as a bridge to analyze textual instruction and add the analysis results to the vision encoder as guidance. With the guidance of text, the vision encoder can extract text-related features, similar to how humans focus on the most relevant parts of an image when considering a question.
  arXiv  Detail & Related papers  (2024-09-15T00:38:34Z)
• Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss [3.435381469869212]
  This paper presents an innovative regression framework for Sentence-BERT STS tasks. It proposes two simple yet effective loss functions: Translated ReLU and Smooth K2 Loss. Experimental results demonstrate that our method achieves convincing performance across seven established STS benchmarks.
  arXiv  Detail & Related papers  (2024-06-08T02:52:43Z)
• Linguistically Conditioned Semantic Textual Similarity [6.049872961766425]
  We reannotate the C-STS validation set and observe annotator discrepancy on 55% of the instances resulting from the annotation errors in the original label. We present an automatic error identification pipeline that is able to identify annotation errors from the CSTS data with over 80% F1 score. We propose a new method that largely improves the performance over baselines on the C-STS data by training the models with the answers.
  arXiv  Detail & Related papers  (2024-06-06T01:23:45Z)
• Text Encoders Lack Knowledge: Leveraging Generative LLMs for Domain-Specific Semantic Textual Similarity [2.861144046639872]
  We show that semantic textual similarity (STS) can be cast as a text generation problem while maintaining strong performance on multiple benchmarks. We show generative LLMs significantly outperform existing encoder-based STS models when characterizing the semantic similarity between two texts. Our results suggest generative language models with STS-specific prompting strategies achieve state-of-the-art performance in complex, domain-specific STS tasks.
  arXiv  Detail & Related papers  (2023-09-12T19:32:45Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• Tracking Objects and Activities with Attention for Temporal Sentence Grounding [51.416914256782505]
  Temporal sentence (TSG) aims to localize the temporal segment which is semantically aligned with a natural language query in an untrimmed segment. We propose a novel Temporal Sentence Tracking Network (TSTNet), which contains (A) a Cross-modal Targets Generator to generate multi-modal and search space, and (B) a Temporal Sentence Tracker to track multi-modal targets' behavior and to predict query-related segment.
  arXiv  Detail & Related papers  (2023-02-21T16:42:52Z)
• SPTS v2: Single-Point Scene Text Spotting [146.98118405786445]
  New framework, SPTS v2, allows us to train high-performing text-spotting models using a single-point annotation. Tests show SPTS v2 can outperform previous state-of-the-art single-point text spotters with fewer parameters. Experiments suggest a potential preference for single-point representation in scene text spotting.
  arXiv  Detail & Related papers  (2023-01-04T14:20:14Z)
• Unsupervised Sentence Textual Similarity with Compositional Phrase Semantics [8.729329792251578]
  Measuring Sentence Textual Similarity (STS) is a classic task that can be applied to many downstream NLP applications. In this paper, we focus on unsupervised STS that works on various domains but only requires minimal data and computational resources.
  arXiv  Detail & Related papers  (2022-10-05T14:14:04Z)
• Composable Text Controls in Latent Space with ODEs [97.12426987887021]
  This paper proposes a new efficient approach for composable text operations in the compact latent space of text. By connecting pretrained LMs to the latent space through efficient adaption, we then decode the sampled vectors into desired text sequences. Experiments show that composing those operators within our approach manages to generate or edit high-quality text.
  arXiv  Detail & Related papers  (2022-08-01T06:51:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.