TorontoCL at CMCL 2021 Shared Task: RoBERTa with Multi-Stage Fine-Tuning for Eye-Tracking Prediction

• URL: http://arxiv.org/abs/2104.07244v1
• Date: Thu, 15 Apr 2021 05:29:13 GMT
• Title: TorontoCL at CMCL 2021 Shared Task: RoBERTa with Multi-Stage Fine-Tuning for Eye-Tracking Prediction
• Authors: Bai Li, Frank Rudzicz
• Abstract summary: We describe our submission to the CMCL 2021 shared task on predicting human reading patterns. Our model uses RoBERTa with a regression layer to predict 5 eye-tracking features. Our final submission achieves a MAE score of 3.929, ranking 3rd place out of 13 teams that participated in this task.
• Score: 25.99947358445936
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Eye movement data during reading is a useful source of information for understanding language comprehension processes. In this paper, we describe our submission to the CMCL 2021 shared task on predicting human reading patterns. Our model uses RoBERTa with a regression layer to predict 5 eye-tracking features. We train the model in two stages: we first fine-tune on the Provo corpus (another eye-tracking dataset), then fine-tune on the task data. We compare different Transformer models and apply ensembling methods to improve the performance. Our final submission achieves a MAE score of 3.929, ranking 3rd place out of 13 teams that participated in this shared task.

Related papers

• Multi-Task Learning for Front-End Text Processing in TTS [15.62497569424995]
  We propose a multi-task learning (MTL) model for jointly performing three tasks that are commonly solved in a text-to-speech front-end. Our framework utilizes a tree-like structure with a trunk that learns shared representations, followed by separate task-specific heads.
  arXiv  Detail & Related papers  (2024-01-12T02:13:21Z)
• IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks [124.90137528319273]
  In this paper, we present IMProv, a generative model that is able to in-context learn visual tasks from multimodal prompts. We train a masked generative transformer on a new dataset of figures from computer vision papers and their associated captions. During inference time, we prompt the model with text and/or image task example(s) and have the model inpaint the corresponding output.
  arXiv  Detail & Related papers  (2023-12-04T09:48:29Z)
• Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching [63.88319217738223]
  We present Matcher, a novel perception paradigm that utilizes off-the-shelf vision foundation models to address various perception tasks. Matcher demonstrates impressive generalization performance across various segmentation tasks, all without training. Our results further showcase the open-world generality and flexibility of Matcher when applied to images in the wild.
  arXiv  Detail & Related papers  (2023-05-22T17:59:43Z)
• Multi-Level Contrastive Learning for Dense Prediction Task [59.591755258395594]
  We present Multi-Level Contrastive Learning for Dense Prediction Task (MCL), an efficient self-supervised method for learning region-level feature representation for dense prediction tasks. Our method is motivated by the three key factors in detection: localization, scale consistency and recognition. Our method consistently outperforms the recent state-of-the-art methods on various datasets with significant margins.
  arXiv  Detail & Related papers  (2023-04-04T17:59:04Z)
• Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs [37.754787051387034]
  We propose a representation learning framework called breakpoint modeling. Our approach trains models in an efficient and end-to-end fashion to build intermediate representations. We show the benefit of our main breakpoint transformer, based on T5, over conventional representation learning approaches.
  arXiv  Detail & Related papers  (2022-11-15T07:28:14Z)
• Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5 [50.574918785575655]
  We compare sequential fine-tuning with a model for multi-task learning in the context of boosting performance on two tasks. Our results show that while sequential multi-task learning can be tuned to be good at the first of two target tasks, it performs less well on the second and additionally struggles with overfitting.
  arXiv  Detail & Related papers  (2022-10-31T13:26:08Z)
• Unifying Language Learning Paradigms [96.35981503087567]
  We present a unified framework for pre-training models that are universally effective across datasets and setups. We show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
  arXiv  Detail & Related papers  (2022-05-10T19:32:20Z)
• Team \'UFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models [9.087729124428467]
  We describe our systems for the CMCL 2022 shared task on predicting eye-tracking information. Our submissions achieved an average MAE of 5.72 and ranked 5th in the shared task.
  arXiv  Detail & Related papers  (2022-04-11T10:43:34Z)
• Zero Shot Crosslingual Eye-Tracking Data Prediction using Multilingual Transformer Models [0.0]
  We describe our submission to the CMCL 2022 shared task on predicting human reading patterns for multi-lingual dataset. Our model uses text representations from transformers and some hand engineered features with a regression layer on top to predict statistical measures of mean and standard deviation. We train an end to end model to extract meaningful information from different languages and test our model on two seperate datasets.
  arXiv  Detail & Related papers  (2022-03-30T17:11:48Z)
• Netmarble AI Center's WMT21 Automatic Post-Editing Shared Task Submission [6.043109546012043]
  This paper describes Netmarble's submission to WMT21 Automatic Post-Editing (APE) Shared Task for the English-German language pair. Facebook Fair's WMT19 news translation model was chosen to engage the large and powerful pre-trained neural networks. For better performance, we leverage external translations as augmented machine translation (MT) during the post-training and fine-tuning.
  arXiv  Detail & Related papers  (2021-09-14T08:21:18Z)
• KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation [100.79870384880333]
  We propose a knowledge-grounded pre-training (KGPT) to generate knowledge-enriched text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under zero-shot setting, our model achieves over 30 ROUGE-L on WebNLG while all other baselines fail.
  arXiv  Detail & Related papers  (2020-10-05T19:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.