Where does In-context Translation Happen in Large Language Models

• URL: http://arxiv.org/abs/2403.04510v1
• Date: Thu, 7 Mar 2024 14:12:41 GMT
• Title: Where does In-context Translation Happen in Large Language Models
• Authors: Suzanna Sia, David Mueller, Kevin Duh
• Abstract summary: We characterize the region where large language models transition from in-text learners to translation models. We demonstrate evidence of a "task recognition" point where the translation task is encoded into the input representations and attention to context is no longer necessary.
• Score: 18.379840329713407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised large language models have demonstrated the ability to perform Machine Translation (MT) via in-context learning, but little is known about where the model performs the task with respect to prompt instructions and demonstration examples. In this work, we attempt to characterize the region where large language models transition from in-context learners to translation models. Through a series of layer-wise context-masking experiments on \textsc{GPTNeo2.7B}, \textsc{Bloom3B}, \textsc{Llama7b} and \textsc{Llama7b-chat}, we demonstrate evidence of a "task recognition" point where the translation task is encoded into the input representations and attention to context is no longer necessary. We further observe correspondence between the low performance when masking out entire layers, and the task recognition layers. Taking advantage of this redundancy results in 45\% computational savings when prompting with 5 examples, and task recognition achieved at layer 14 / 32. Our layer-wise fine-tuning experiments indicate that the most effective layers for MT fine-tuning are the layers critical to task recognition.

Related papers

• Probing Context Localization of Polysemous Words in Pre-trained Language Model Sub-Layers [12.610445666406898]
  We investigate the degree of contextualization encoded in the fine-grained sub-layer representations of a Pre-trained Language Model (PLM) To identify the main contributions of sub-layers to contextualisation, we first extract the sub-layer representations of polysemous words in minimally different sentence pairs. We also try to empirically localize the strength of contextualization information encoded in these sub-layer representations.
  arXiv  Detail & Related papers  (2024-09-21T10:42:07Z)
• VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models [76.94378391979228]
  We introduce a new, more demanding task known as Interleaved Image-Text (IITC) This task challenges models to discern and disregard superfluous elements in both images and text to accurately answer questions. In support of this task, we further craft a new VEGA dataset, tailored for the IITC task on scientific content, and devised a subtask, Image-Text Association (ITA)
  arXiv  Detail & Related papers  (2024-06-14T17:59:40Z)
• Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning [68.43706033424378]
  This study introduces an innovative method designed to increase in-context text length in large language models (MLLMs) efficiently. We present Visualized In-Context Text Processing (VisInContext), which processes long in-context text using visual tokens. This technique significantly reduces GPU memory usage and floating point operations (FLOPs) for both training and inferenceing stage.
  arXiv  Detail & Related papers  (2024-06-04T17:59:25Z)
• Can Large Language Models Understand Context? [17.196362853457412]
  This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. As LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings.
  arXiv  Detail & Related papers  (2024-02-01T18:55:29Z)
• Prompting Large Language Model for Machine Translation: A Case Study [87.88120385000666]
  We offer a systematic study on prompting strategies for machine translation. We examine factors for prompt template and demonstration example selection. We explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning.
  arXiv  Detail & Related papers  (2023-01-17T18:32:06Z)
• Beyond Bounding Box: Multimodal Knowledge Learning for Object Detection [3.785123406103386]
  We take advantage of language prompt to introduce effective and unbiased linguistic supervision into object detection. We propose a new mechanism called multimodal knowledge learning (textbfMKL), which is required to learn knowledge from language supervision.
  arXiv  Detail & Related papers  (2022-05-09T07:03:30Z)
• When Does Translation Require Context? A Data-driven, Multilingual Exploration [71.43817945875433]
  proper handling of discourse significantly contributes to the quality of machine translation (MT) Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation. We develop the Multilingual Discourse-Aware benchmark, a series of taggers that identify and evaluate model performance on discourse phenomena.
  arXiv  Detail & Related papers  (2021-09-15T17:29:30Z)
• Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little [74.49773960145681]
  A possible explanation for the impressive performance of masked language model (MLM)-training is that such models have learned to represent the syntactic structures prevalent in NLP pipelines. In this paper, we propose a different explanation: pre-trains succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. Our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.
  arXiv  Detail & Related papers  (2021-04-14T06:30:36Z)
• Larger-Context Tagging: When and Why Does It Work? [55.407651696813396]
  We focus on investigating when and why the larger-context training, as a general strategy, can work. We set up a testbed based on four tagging tasks and thirteen datasets.
  arXiv  Detail & Related papers  (2021-04-09T15:35:30Z)
• Quantifying the Contextualization of Word Representations with Semantic Class Probing [8.401007663676214]
  Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well. We quantify the amount of contextualization, i.e., how well words are interpreted in context, by studying the extent to which semantic classes of a word can be inferred from its contextualized embeddings.
  arXiv  Detail & Related papers  (2020-04-25T17:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.