A Cross-Task Analysis of Text Span Representations

• URL: http://arxiv.org/abs/2006.03866v1
• Date: Sat, 6 Jun 2020 13:37:51 GMT
• Title: A Cross-Task Analysis of Text Span Representations
• Authors: Shubham Toshniwal, Haoyue Shi, Bowen Shi, Lingyu Gao, Karen Livescu, Kevin Gimpel
• Abstract summary: We find that the optimal span representation varies by task, and can also vary within different facets of individual tasks. We also find that the choice of span representation has a bigger impact with a fixed pretrained encoder than with a fine-tuned encoder.
• Score: 52.28565379517174
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution. While extensive research has focused on functional architectures for representing words and sentences, there is less work on representing arbitrary spans of text within sentences. In this paper, we conduct a comprehensive empirical evaluation of six span representation methods using eight pretrained language representation models across six tasks, including two tasks that we introduce. We find that, although some simple span representations are fairly reliable across tasks, in general the optimal span representation varies by task, and can also vary within different facets of individual tasks. We also find that the choice of span representation has a bigger impact with a fixed pretrained encoder than with a fine-tuned encoder.

Related papers

• Multilingual Multi-Figurative Language Detection [14.799109368073548]
  figurative language understanding is highly understudied in a multilingual setting. We introduce multilingual multi-figurative language modelling, and provide a benchmark for sentence-level figurative language detection. We develop a framework for figurative language detection based on template-based prompt learning.
  arXiv  Detail & Related papers  (2023-05-31T18:52:41Z)
• Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefixes [22.661527526471996]
  Fine-tuning a large pre-trained language model for each downstream task causes computational burdens. We propose a prefix-based method to learn the fixed text representations with source tasks.
  arXiv  Detail & Related papers  (2023-05-22T21:31:03Z)
• Universal Multimodal Representation for Language Understanding [110.98786673598015]
  This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively.
  arXiv  Detail & Related papers  (2023-01-09T13:54:11Z)
• Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5 [50.574918785575655]
  We compare sequential fine-tuning with a model for multi-task learning in the context of boosting performance on two tasks. Our results show that while sequential multi-task learning can be tuned to be good at the first of two target tasks, it performs less well on the second and additionally struggles with overfitting.
  arXiv  Detail & Related papers  (2022-10-31T13:26:08Z)
• Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning [54.66399120084227]
  Recent state-of-the-art neural text matching models (PLMs) are hard to generalize to different tasks. We adopt a specialization-generalization training strategy and refer to it as Match-Prompt. In specialization stage, descriptions of different matching tasks are mapped to only a few prompt tokens. In generalization stage, text matching model explores the essential matching signals by being trained on diverse multiple matching tasks.
  arXiv  Detail & Related papers  (2022-04-06T11:01:08Z)
• Grad2Task: Improved Few-shot Text Classification Using Gradients for Task Representation [24.488427641442694]
  We propose a novel conditional neural process-based approach for few-shot text classification. Our key idea is to represent each task using gradient information from a base model. Our approach outperforms traditional fine-tuning, sequential transfer learning, and state-of-the-art meta learning approaches.
  arXiv  Detail & Related papers  (2022-01-27T15:29:30Z)
• Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition [60.36540008537054]
  In this work, we excavate the implicit task, character counting within the traditional text recognition, without additional labor annotation cost. We design a two-branch reciprocal feature learning framework in order to adequately utilize the features from both the tasks. Experiments on 7 benchmarks show the advantages of the proposed methods in both text recognition and the new-built character counting tasks.
  arXiv  Detail & Related papers  (2021-05-13T12:27:35Z)
• Intrinsic Probing through Dimension Selection [69.52439198455438]
  Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted.
  arXiv  Detail & Related papers  (2020-10-06T15:21:08Z)
• Evaluating Multimodal Representations on Visual Semantic Textual Similarity [22.835699807110018]
  We present a novel task, Visual Semantic Textual Similarity (vSTS), where such inference ability can be tested directly. Our experiments using simple multimodal representations show that the addition of image representations produces better inference, compared to text-only representations. Our work shows, for the first time, the successful contribution of visual information to textual inference, with ample room for more complex multimodal representation options.
  arXiv  Detail & Related papers  (2020-04-04T09:03:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.