What Makes Language Models Good-enough?

• URL: http://arxiv.org/abs/2406.03666v1
• Date: Thu, 6 Jun 2024 00:51:28 GMT
• Title: What Makes Language Models Good-enough?
• Authors: Daiki Asami, Saku Sugawara,
• Abstract summary: Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural features make language models learn human-like good-enough language processing.
• Score: 11.763229353978321
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural features make language models learn human-like good-enough language processing. We focus on the number of layers and self-attention heads in Transformers. We create a good-enough language processing (GELP) evaluation dataset (7,680 examples), which is designed to test the effects of two plausibility types, eight construction types, and three degrees of memory cost on language processing. To annotate GELP, we first conduct a crowdsourcing experiment whose design follows prior psycholinguistic studies. Our model evaluation against the annotated GELP then reveals that the full model as well as models with fewer layers and/or self-attention heads exhibit a good-enough performance. This result suggests that models with shallower depth and fewer heads can learn good-enough language processing.

Related papers

• Dual-Layer Training and Decoding of Large Language Model with Simultaneously Thinking and Speaking [8.02728252625147]
  Large Language Model can reasonably understand and generate human expressions but may lack thorough thinking and reasoning mechanisms. In this paper, we are motivated by the cognitive mechanism in the natural world, and design a novel model architecture called TaS. We train the language model by the thoughts-augmented data and successfully let the thinking layer automatically generate reasonable thoughts and finally output more reasonable responses.
  arXiv  Detail & Related papers  (2024-09-18T15:32:48Z)
• Revisiting Topic-Guided Language Models [20.21486464604549]
  We study four topic-guided language models and two baselines, evaluating the held-out predictive performance of each model on four corpora. We find that none of these methods outperform a standard LSTM language model baseline, and most fail to learn good topics.
  arXiv  Detail & Related papers  (2023-12-04T20:33:24Z)
• Robustifying Language Models with Test-Time Adaptation [17.96043752001886]
  Large-scale language models achieved state-of-the-art performance over a number of language tasks. They fail on adversarial language examples, which are sentences optimized to fool the language models but with similar semantic meanings for humans. We show that we can reverse many language adversarial attacks by adapting the input sentence with predictions from masked words.
  arXiv  Detail & Related papers  (2023-10-29T22:37:54Z)
• Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention [58.817405319722596]
  This work compares the self-attention of several large language models (LLMs) in different sizes to assess the effect of scaling and instruction tuning on language perception. Results show that scaling enhances the human resemblance and improves the effective attention by reducing the trivial pattern reliance, while instruction tuning does not. We also find that current LLMs are consistently closer to non-native than native speakers in attention, suggesting a sub-optimal language perception of all models.
  arXiv  Detail & Related papers  (2023-10-29T17:16:40Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• Chain of Hindsight Aligns Language Models with Feedback [62.68665658130472]
  We propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors.
  arXiv  Detail & Related papers  (2023-02-06T10:28:16Z)
• Training Language Models with Natural Language Feedback [51.36137482891037]
  We learn from language feedback on model outputs using a three-step learning algorithm. In synthetic experiments, we first evaluate whether language models accurately incorporate feedback to produce refinements. Using only 100 samples of human-written feedback, our learning algorithm finetunes a GPT-3 model to roughly human-level summarization.
  arXiv  Detail & Related papers  (2022-04-29T15:06:58Z)
• Estimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation [0.0]
  We use a crowd-authored dialogue corpus to fine-tune six different language generation models. Two of these models incorporate multi-task learning and use subjective ratings of lines as part of an explicit learning goal. A human evaluation of the generated dialogue lines reveals that utterances generated by the multi-tasking models were subjectively rated as the most typical, most moving the conversation forward, and least offensive.
  arXiv  Detail & Related papers  (2021-04-12T06:33:16Z)
• Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition [80.446770909975]
  Linguistic knowledge is of great benefit to scene text recognition. How to effectively model linguistic rules in end-to-end deep networks remains a research challenge. We propose an autonomous, bidirectional and iterative ABINet for scene text recognition.
  arXiv  Detail & Related papers  (2021-03-11T06:47:45Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.