Span-Aggregatable, Contextualized Word Embeddings for Effective Phrase Mining

• URL: http://arxiv.org/abs/2405.07263v1
• Date: Sun, 12 May 2024 12:08:05 GMT
• Title: Span-Aggregatable, Contextualized Word Embeddings for Effective Phrase Mining
• Authors: Eyal Orbach, Lev Haikin, Nelly David, Avi Faizakof,
• Abstract summary: We show that when target phrases reside inside noisy context, representing the full sentence with a single dense vector is not sufficient for effective phrase retrieval. We show that this technique is much more effective for phrase mining, yet requires considerable compute to obtain useful span representations.
• Score: 0.22499166814992438
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Dense vector representations for sentences made significant progress in recent years as can be seen on sentence similarity tasks. Real-world phrase retrieval applications, on the other hand, still encounter challenges for effective use of dense representations. We show that when target phrases reside inside noisy context, representing the full sentence with a single dense vector, is not sufficient for effective phrase retrieval. We therefore look into the notion of representing multiple, sub-sentence, consecutive word spans, each with its own dense vector. We show that this technique is much more effective for phrase mining, yet requires considerable compute to obtain useful span representations. Accordingly, we make an argument for contextualized word/token embeddings that can be aggregated for arbitrary word spans while maintaining the span's semantic meaning. We introduce a modification to the common contrastive loss used for sentence embeddings that encourages word embeddings to have this property. To demonstrate the effect of this method we present a dataset based on the STS-B dataset with additional generated text, that requires finding the best matching paraphrase residing in a larger context and report the degree of similarity to the origin phrase. We demonstrate on this dataset, how our proposed method can achieve better results without significant increase to compute.

Related papers

• Relational Sentence Embedding for Flexible Semantic Matching [86.21393054423355]
  We present Sentence Embedding (RSE), a new paradigm to discover further the potential of sentence embeddings. RSE is effective and flexible in modeling sentence relations and outperforms a series of state-of-the-art embedding methods.
  arXiv  Detail & Related papers  (2022-12-17T05:25:17Z)
• Searching for Discriminative Words in Multidimensional Continuous Feature Space [0.0]
  We propose a novel method to extract discriminative keywords from documents. We show how different discriminative metrics influence the overall results. We conclude that word feature vectors can substantially improve the topical inference of documents' meaning.
  arXiv  Detail & Related papers  (2022-11-26T18:05:11Z)
• Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences [69.3939291118954]
  This paper reports research on a set of comprehensive clustering and network analyses targeting sentence and sub-sentence embedding spaces. Results show that one method generates the most clusterable embeddings. In general, the embeddings of span sub-sentences have better clustering properties than the original sentences.
  arXiv  Detail & Related papers  (2021-10-02T00:47:35Z)
• Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration [25.159601117722936]
  We propose a contrastive fine-tuning objective that enables BERT to produce more powerful phrase embeddings. Our approach relies on a dataset of diverse phrasal paraphrases, which is automatically generated using a paraphrase generation model. As a case study, we show that Phrase-BERT embeddings can be easily integrated with a simple autoencoder to build a phrase-based neural topic model.
  arXiv  Detail & Related papers  (2021-09-13T20:31:57Z)
• Improving Paraphrase Detection with the Adversarial Paraphrasing Task [0.0]
  Paraphrasing datasets currently rely on a sense of paraphrase based on word overlap and syntax. We introduce a new adversarial method of dataset creation for paraphrase identification: the Adversarial Paraphrasing Task (APT) APT asks participants to generate semantically equivalent (in the sense of mutually implicative) but lexically and syntactically disparate paraphrases.
  arXiv  Detail & Related papers  (2021-06-14T18:15:20Z)
• An Iterative Contextualization Algorithm with Second-Order Attention [0.40611352512781856]
  We show how to combine the representations of words that make up a sentence into a cohesive whole. Our algorithm starts with a presumably erroneous value of the context, and adjusts this value with respect to the tokens at hand. Our models report strong results in several well-known text classification tasks.
  arXiv  Detail & Related papers  (2021-03-03T05:34:50Z)
• Match-Ignition: Plugging PageRank into Transformer for Long-form Text Matching [66.71886789848472]
  We propose a novel hierarchical noise filtering model, namely Match-Ignition, to tackle the effectiveness and efficiency problem. The basic idea is to plug the well-known PageRank algorithm into the Transformer, to identify and filter both sentence and word level noisy information. Noisy sentences are usually easy to detect because the sentence is the basic unit of a long-form text, so we directly use PageRank to filter such information.
  arXiv  Detail & Related papers  (2021-01-16T10:34:03Z)
• R$^2$-Net: Relation of Relation Learning Network for Sentence Semantic Matching [58.72111690643359]
  We propose a Relation of Relation Learning Network (R2-Net) for sentence semantic matching. We first employ BERT to encode the input sentences from a global perspective. Then a CNN-based encoder is designed to capture keywords and phrase information from a local perspective. To fully leverage labels for better relation information extraction, we introduce a self-supervised relation of relation classification task.
  arXiv  Detail & Related papers  (2020-12-16T13:11:30Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Unsupervised Summarization by Jointly Extracting Sentences and Keywords [12.387378783627762]
  RepRank is an unsupervised graph-based ranking model for extractive multi-document summarization. We show that salient sentences and keywords can be extracted in a joint and mutual reinforcement process using our learned representations. Experiment results with multiple benchmark datasets show that RepRank achieved the best or comparable performance in ROUGE.
  arXiv  Detail & Related papers  (2020-09-16T05:58:00Z)
• Extractive Summarization as Text Matching [123.09816729675838]
  This paper creates a paradigm shift with regard to the way we build neural extractive summarization systems. We formulate the extractive summarization task as a semantic text matching problem. We have driven the state-of-the-art extractive result on CNN/DailyMail to a new level (44.41 in ROUGE-1)
  arXiv  Detail & Related papers  (2020-04-19T08:27:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.