The Impact of Token Granularity on the Predictive Power of Language Model Surprisal

• URL: http://arxiv.org/abs/2412.11940v1
• Date: Mon, 16 Dec 2024 16:24:58 GMT
• Title: The Impact of Token Granularity on the Predictive Power of Language Model Surprisal
• Authors: Byung-Doh Oh, William Schuler,
• Abstract summary: One factor that has been overlooked in cognitive modeling is the granularity of subword tokens. Experiments with naturalistic reading times reveal a substantial influence of token granularity on surprisal. On garden-path constructions, language models trained on coarser-grained tokens generally assigned higher surprisal to critical regions.
• Score: 15.073507986272027
• License:
• Abstract: Word-by-word language model surprisal is often used to model the incremental processing of human readers, which raises questions about how various choices in language modeling influence its predictive power. One factor that has been overlooked in cognitive modeling is the granularity of subword tokens, which explicitly encodes information about word length and frequency, and ultimately influences the quality of vector representations that are learned. This paper presents experiments that manipulate the token granularity and evaluate its impact on the ability of surprisal to account for processing difficulty of naturalistic text and garden-path constructions. Experiments with naturalistic reading times reveal a substantial influence of token granularity on surprisal, with tokens defined by a vocabulary size of 8,000 resulting in surprisal that is most predictive. In contrast, on garden-path constructions, language models trained on coarser-grained tokens generally assigned higher surprisal to critical regions, suggesting their increased sensitivity to syntax. Taken together, these results suggest a large role of token granularity on the quality of language model surprisal for cognitive modeling.

Related papers

• How does a Language-Specific Tokenizer affect LLMs? [0.36248657646376703]
  The necessity of language-specific tokenizers intuitively appears crucial for effective natural language processing. This study explores how language-specific tokenizers influence the behavior of Large Language Models predominantly trained with English text data.
  arXiv  Detail & Related papers  (2025-02-18T05:54:56Z)
• On the Proper Treatment of Tokenization in Psycholinguistics [53.960910019072436]
  The paper argues that token-level language models should be marginalized into character-level language models before they are used in psycholinguistic studies. We find various focal areas whose surprisal is a better psychometric predictor than the surprisal of the region of interest itself.
  arXiv  Detail & Related papers  (2024-10-03T17:18:03Z)
• Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models [0.0]
  This paper examines the impact of tokenization strategies and vocabulary sizes on the performance of Arabic language models. Our study uncovers limited impacts of vocabulary size on model performance while keeping the model size unchanged. Paper's recommendations include refining tokenization strategies to address dialect challenges, enhancing model robustness across diverse linguistic contexts, and expanding datasets to encompass the rich dialect based Arabic.
  arXiv  Detail & Related papers  (2024-03-17T07:44:44Z)
• Language Model Behavior: A Comprehensive Survey [5.663056267168211]
  We discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases.
  arXiv  Detail & Related papers  (2023-03-20T23:54:26Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Quark: Controllable Text Generation with Reinforced Unlearning [68.07749519374089]
  Large-scale language models often learn behaviors that are misaligned with user expectations. We introduce Quantized Reward Konditioning (Quark), an algorithm for optimizing a reward function that quantifies an (un)wanted property. For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods.
  arXiv  Detail & Related papers  (2022-05-26T21:11:51Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• Evaluating Models of Robust Word Recognition with Serial Reproduction [8.17947290421835]
  We compare several broad-coverage probabilistic generative language models in their ability to capture human linguistic expectations. We find that those models that make use of abstract representations of preceding linguistic context best predict the changes made by people in the course of serial reproduction.
  arXiv  Detail & Related papers  (2021-01-24T20:16:12Z)
• Do Language Embeddings Capture Scales? [54.1633257459927]
  We show that pretrained language models capture a significant amount of information about the scalar magnitudes of objects. We identify contextual information in pre-training and numeracy as two key factors affecting their performance.
  arXiv  Detail & Related papers  (2020-10-11T21:11:09Z)
• Linguistic Profiling of a Neural Language Model [1.0552465253379135]
  We investigate the linguistic knowledge learned by a Neural Language Model (NLM) before and after a fine-tuning process. We show that BERT is able to encode a wide range of linguistic characteristics, but it tends to lose this information when trained on specific downstream tasks.
  arXiv  Detail & Related papers  (2020-10-05T09:09:01Z)
• Limits of Detecting Text Generated by Large-Scale Language Models [65.46403462928319]
  Some consider large-scale language models that can generate long and coherent pieces of text as dangerous, since they may be used in misinformation campaigns. Here we formulate large-scale language model output detection as a hypothesis testing problem to classify text as genuine or generated.
  arXiv  Detail & Related papers  (2020-02-09T19:53:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.