Probing Representations for Document-level Event Extraction

• URL: http://arxiv.org/abs/2310.15316v1
• Date: Mon, 23 Oct 2023 19:33:04 GMT
• Title: Probing Representations for Document-level Event Extraction
• Authors: Barry Wang and Xinya Du and Claire Cardie
• Abstract summary: This work is the first to apply the probing paradigm to representations learned for document-level information extraction. We designed eight embedding probes to analyze surface, semantic, and event-understanding capabilities relevant to document-level event extraction. We found that trained encoders from these models yield embeddings that can modestly improve argument detections and labeling but only slightly enhance event-level tasks.
• Score: 30.523959637364484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The probing classifiers framework has been employed for interpreting deep neural network models for a variety of natural language processing (NLP) applications. Studies, however, have largely focused on sentencelevel NLP tasks. This work is the first to apply the probing paradigm to representations learned for document-level information extraction (IE). We designed eight embedding probes to analyze surface, semantic, and event-understanding capabilities relevant to document-level event extraction. We apply them to the representations acquired by learning models from three different LLM-based document-level IE approaches on a standard dataset. We found that trained encoders from these models yield embeddings that can modestly improve argument detections and labeling but only slightly enhance event-level tasks, albeit trade-offs in information helpful for coherence and event-type prediction. We further found that encoder models struggle with document length and cross-sentence discourse.

Related papers

• Pointer-Guided Pre-Training: Infusing Large Language Models with Paragraph-Level Contextual Awareness [3.2925222641796554]
  "pointer-guided segment ordering" (SO) is a novel pre-training technique aimed at enhancing the contextual understanding of paragraph-level text representations. Our experiments show that pointer-guided pre-training significantly enhances the model's ability to understand complex document structures.
  arXiv  Detail & Related papers  (2024-06-06T15:17:51Z)
• SLCNN: Sentence-Level Convolutional Neural Network for Text Classification [0.0]
  Convolutional neural network (CNN) has shown remarkable success in the task of text classification. New baseline models have been studied for text classification using CNN. Results have shown that the proposed models have better performance, particularly in the longer documents.
  arXiv  Detail & Related papers  (2023-01-27T13:16:02Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Generalization Properties of Retrieval-based Models [50.35325326050263]
  Retrieval-based machine learning methods have enjoyed success on a wide range of problems. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. We present a formal treatment of retrieval-based models to characterize their generalization ability.
  arXiv  Detail & Related papers  (2022-10-06T00:33:01Z)
• Recitation-Augmented Language Models [85.30591349383849]
  We show that RECITE is a powerful paradigm for knowledge-intensive NLP tasks. Specifically, we show that by utilizing recitation as the intermediate step, a recite-and-answer scheme can achieve new state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-10-04T00:49:20Z)
• A Unified Understanding of Deep NLP Models for Text Classification [88.35418976241057]
  We have developed a visual analysis tool, DeepNLPVis, to enable a unified understanding of NLP models for text classification. The key idea is a mutual information-based measure, which provides quantitative explanations on how each layer of a model maintains the information of input words in a sample. A multi-level visualization, which consists of a corpus-level, a sample-level, and a word-level visualization, supports the analysis from the overall training set to individual samples.
  arXiv  Detail & Related papers  (2022-06-19T08:55:07Z)
• Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
  We propose methods to facilitate identification of training data artifacts. We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data. We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
  arXiv  Detail & Related papers  (2021-07-01T09:26:13Z)
• Robust Document Representations using Latent Topics and Metadata [17.306088038339336]
  We propose a novel approach to fine-tuning a pre-trained neural language model for document classification problems. We generate document representations that capture both text and metadata artifacts in a task manner. Our solution also incorporates metadata explicitly rather than just augmenting them with text.
  arXiv  Detail & Related papers  (2020-10-23T21:52:38Z)
• SPECTER: Document-level Representation Learning using Citation-informed Transformers [51.048515757909215]
  SPECTER generates document-level embedding of scientific documents based on pretraining a Transformer language model. We introduce SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction to document classification and recommendation.
  arXiv  Detail & Related papers  (2020-04-15T16:05:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.