NewsEmbed: Modeling News through Pre-trained DocumentRepresentations

• URL: http://arxiv.org/abs/2106.00590v1
• Date: Tue, 1 Jun 2021 15:59:40 GMT
• Title: NewsEmbed: Modeling News through Pre-trained DocumentRepresentations
• Authors: Jialu Liu, Tianqi Liu, Cong Yu
• Abstract summary: We propose a novel approach to mine semantically-relevant fresh documents, and their topic labels, with little human supervision. We show that the proposed approach can provide billions of high quality organic training examples and can be naturally extended to multilingual setting.
• Score: 5.007237648361745
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effectively modeling text-rich fresh content such as news articles at document-level is a challenging problem. To ensure a content-based model generalize well to a broad range of applications, it is critical to have a training dataset that is large beyond the scale of human labels while achieving desired quality. In this work, we address those two challenges by proposing a novel approach to mine semantically-relevant fresh documents, and their topic labels, with little human supervision. Meanwhile, we design a multitask model called NewsEmbed that alternatively trains a contrastive learning with a multi-label classification to derive a universal document encoder. We show that the proposed approach can provide billions of high quality organic training examples and can be naturally extended to multilingual setting where texts in different languages are encoded in the same semantic space. We experimentally demonstrate NewsEmbed's competitive performance across multiple natural language understanding tasks, both supervised and unsupervised.

Related papers

• Text-Guided Mixup Towards Long-Tailed Image Categorization [7.207351201912651]
  In many real-world applications, the frequency distribution of class labels for training data can exhibit a long-tailed distribution. We propose a novel text-guided mixup technique that takes advantage of the semantic relations between classes recognized by the pre-trained text encoder.
  arXiv  Detail & Related papers  (2024-09-05T14:37:43Z)
• Universal Cross-Lingual Text Classification [0.3958317527488535]
  This research proposes a novel perspective on Universal Cross-Lingual Text Classification. Our approach involves blending supervised data from different languages during training to create a universal model. The primary goal is to enhance label and language coverage, aiming for a label set that represents a union of labels from various languages.
  arXiv  Detail & Related papers  (2024-06-16T17:58:29Z)
• Meta-Task Prompting Elicits Embeddings from Large Language Models [54.757445048329735]
  We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation. We generate high-quality sentence embeddings from Large Language Models without the need for model fine-tuning. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.
  arXiv  Detail & Related papers  (2024-02-28T16:35:52Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets [56.018551958004814]
  This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources. Large-scale datasets with noisy image-text pairs provide a sub-optimal source of supervision. We propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component.
  arXiv  Detail & Related papers  (2021-11-24T19:00:05Z)
• Automatic Document Sketching: Generating Drafts from Analogous Texts [44.626645471195495]
  We introduce a new task, document sketching, which involves generating entire draft documents for the writer to review and revise. These drafts are built from sets of documents that overlap in form - sharing large segments of potentially reusable text - while diverging in content. We investigate the application of weakly supervised methods, including use of a transformer-based mixture of experts, together with reinforcement learning.
  arXiv  Detail & Related papers  (2021-06-14T06:46:06Z)
• Minimally-Supervised Structure-Rich Text Categorization via Learning on Text-Rich Networks [61.23408995934415]
  We propose a novel framework for minimally supervised categorization by learning from the text-rich network. Specifically, we jointly train two modules with different inductive biases -- a text analysis module for text understanding and a network learning module for class-discriminative, scalable network learning. Our experiments show that given only three seed documents per category, our framework can achieve an accuracy of about 92%.
  arXiv  Detail & Related papers  (2021-02-23T04:14:34Z)
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations [4.36561468436181]
  We present DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. Our approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. Our code and pretrained models are publicly available and can be easily adapted to new domains or used to embed unseen text.
  arXiv  Detail & Related papers  (2020-06-05T20:00:28Z)
• Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation [50.01708049531156]
  We focus on a new practical task, document-scale text content manipulation, which is the opposite of text style transfer. In detail, the input is a set of structured records and a reference text for describing another recordset. The output is a summary that accurately describes the partial content in the source recordset with the same writing style of the reference.
  arXiv  Detail & Related papers  (2020-02-24T12:52:10Z)
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer [64.22926988297685]
  Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP) In this paper, we explore the landscape of introducing transfer learning techniques for NLP by a unified framework that converts all text-based language problems into a text-to-text format.
  arXiv  Detail & Related papers  (2019-10-23T17:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.