Text and Code Embeddings by Contrastive Pre-Training

• URL: http://arxiv.org/abs/2201.10005v1
• Date: Mon, 24 Jan 2022 23:36:20 GMT
• Title: Text and Code Embeddings by Contrastive Pre-Training
• Authors: Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, Lilian Weng
• Abstract summary: We show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. Same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities.
• Score: 15.099849247795714
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use cases, varying in dataset choice, training objective and model architecture. In this work, we show that contrastive pre-training on unsupervised data at scale leads to high quality vector representations of text and code. The same unsupervised text embeddings that achieve new state-of-the-art results in linear-probe classification also display impressive semantic search capabilities and sometimes even perform competitively with fine-tuned models. On linear-probe classification accuracy averaging over 7 tasks, our best unsupervised model achieves a relative improvement of 4% and 1.8% over previous best unsupervised and supervised text embedding models respectively. The same text embeddings when evaluated on large-scale semantic search attains a relative improvement of 23.4%, 14.7%, and 10.6% over previous best unsupervised methods on MSMARCO, Natural Questions and TriviaQA benchmarks, respectively. Similarly to text embeddings, we train code embedding models on (text, code) pairs, obtaining a 20.8% relative improvement over prior best work on code search.

Related papers

• Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Sampling [0.0]
  APEX-Embedding-7B is a 7-billion parameter decoder-only text Feature Extraction Model. Our approach employs two training techniques that yield an emergent improvement in factual focus. Based on our evaluations, our model establishes a new state-of-the-art standard in text feature extraction for longer context document retrieval tasks.
  arXiv  Detail & Related papers  (2024-10-08T17:36:48Z)
• Improving embedding with contrastive fine-tuning on small datasets with expert-augmented scores [12.86467344792873]
  The proposed method uses soft labels derived from expert-augmented scores to fine-tune embedding models. The paper evaluates the method using a Q&A dataset from an online shopping website and eight expert models.
  arXiv  Detail & Related papers  (2024-08-19T01:59:25Z)
• Text Quality-Based Pruning for Efficient Training of Language Models [66.66259229732121]
  We propose a novel method for numerically evaluating text quality in large unlabelled NLP datasets. By proposing the text quality metric, the paper establishes a framework to identify and eliminate low-quality text instances. Experimental results over multiple models and datasets demonstrate the efficacy of this approach.
  arXiv  Detail & Related papers  (2024-04-26T18:01:25Z)
• GuideWalk: A Novel Graph-Based Word Embedding for Enhanced Text Classification [0.0]
  The processing of text data requires embedding, a method of translating the content of the text to numeric vectors. A new text embedding approach, namely the Guided Transition Probability Matrix (GTPM) model is proposed. The proposed method is tested with real-world data sets and eight well-known and successful embedding algorithms.
  arXiv  Detail & Related papers  (2024-04-25T18:48:11Z)
• Contrastive Transformer Learning with Proximity Data Generation for Text-Based Person Search [60.626459715780605]
  Given a descriptive text query, text-based person search aims to retrieve the best-matched target person from an image gallery. Such a cross-modal retrieval task is quite challenging due to significant modality gap, fine-grained differences and insufficiency of annotated data. In this paper, we propose a simple yet effective dual Transformer model for text-based person search.
  arXiv  Detail & Related papers  (2023-11-15T16:26:49Z)
• Towards General Text Embeddings with Multi-stage Contrastive Learning [20.803769345818456]
  GTE is a general-purpose text embedding model trained with multi-stage contrastive learning. We train a unified text embedding model by employing contrastive learning over a diverse mixture of datasets from multiple sources.
  arXiv  Detail & Related papers  (2023-08-07T03:52:59Z)
• Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond [52.656743602538825]
  Fine-tuning pre-trained code models incurs a large computational cost. We conduct an experimental study to explore what happens to layer-wise pre-trained representations and their encoded code knowledge during fine-tuning. We propose Telly to efficiently fine-tune pre-trained code models via layer freezing.
  arXiv  Detail & Related papers  (2023-04-11T13:34:13Z)
• Prompt-based Learning for Text Readability Assessment [0.4757470449749875]
  We propose the novel adaptation of a pre-trained seq2seq model for readability assessment. We prove that a seq2seq model can be adapted to discern which text is more difficult from two given texts (pairwise)
  arXiv  Detail & Related papers  (2023-02-25T18:39:59Z)
• JOIST: A Joint Speech and Text Streaming Model For ASR [63.15848310748753]
  We present JOIST, an algorithm to train a streaming, cascaded, encoder end-to-end (E2E) model with both speech-text paired inputs, and text-only unpaired inputs. We find that best text representation for JOIST improves WER across a variety of search and rare-word test sets by 4-14% relative, compared to a model not trained with text.
  arXiv  Detail & Related papers  (2022-10-13T20:59:22Z)
• Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability [74.11825654535895]
  We investigate whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. We find that even on non-text data, the models pre-trained on text converge faster than the randomly models.
  arXiv  Detail & Related papers  (2021-03-12T09:19:14Z)
• Automated Concatenation of Embeddings for Structured Prediction [75.44925576268052]
  We propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model.
  arXiv  Detail & Related papers  (2020-10-10T14:03:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.