What Makes Two Language Models Think Alike?

• URL: http://arxiv.org/abs/2406.12620v2
• Date: Mon, 24 Jun 2024 06:34:35 GMT
• Title: What Makes Two Language Models Think Alike?
• Authors: Jeanne Salle, Louis Jalouzot, Nur Lan, Emmanuel Chemla, Yair Lakretz,
• Abstract summary: We propose a new approach, based on metric-learning encoding models (MLEMs), as a first step to answer this question. MLEMs offer a transparent comparison, by identifying the specific linguistic features responsible for similarities and differences. The approach can straightforwardly be extended to other domains, such as speech and vision, and to other neural systems, including human brains.
• Score: 6.244579327420724
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Do architectural differences significantly affect the way models represent and process language? We propose a new approach, based on metric-learning encoding models (MLEMs), as a first step to answer this question. The approach provides a feature-based comparison of how any two layers of any two models represent linguistic information. We apply the method to BERT, GPT-2 and Mamba. Unlike previous methods, MLEMs offer a transparent comparison, by identifying the specific linguistic features responsible for similarities and differences. More generally, the method uses formal, symbolic descriptions of a domain, and use these to compare neural representations. As such, the approach can straightforwardly be extended to other domains, such as speech and vision, and to other neural systems, including human brains.

Related papers

• Metric-Learning Encoding Models Identify Processing Profiles of Linguistic Features in BERT's Representations [5.893248479095486]
  Metric-Learning Models (MLEMs) are a new approach to understand how neural systems represent the theoretical features of the objects they process. MLEMs can be extended to other domains (e.g. vision) and to other neural systems, such as the human brain.
  arXiv  Detail & Related papers  (2024-02-18T14:57:53Z)
• Pixel Sentence Representation Learning [67.4775296225521]
  In this work, we conceptualize the learning of sentence-level textual semantics as a visual representation learning process. We employ visually-grounded text perturbation methods like typos and word order shuffling, resonating with human cognitive patterns, and enabling perturbation to be perceived as continuous. Our approach is further bolstered by large-scale unsupervised topical alignment training and natural language inference supervision.
  arXiv  Detail & Related papers  (2024-02-13T02:46:45Z)
• Divergences between Language Models and Human Brains [63.405788999891335]
  Recent research has hinted that brain signals can be effectively predicted using internal representations of language models (LMs) We show that there are clear differences in how LMs and humans represent and use language. We identify two domains that are not captured well by LMs: social/emotional intelligence and physical commonsense.
  arXiv  Detail & Related papers  (2023-11-15T19:02:40Z)
• Interpretable Differencing of Machine Learning Models [20.99877540751412]
  We formalize the problem of model differencing as one of predicting a dissimilarity function of two ML models' outputs. A Joint Surrogate Tree (JST) is composed of two conjoined decision tree surrogates for the two models. A JST provides an intuitive representation of differences and places the changes in the context of the models' decision logic.
  arXiv  Detail & Related papers  (2023-06-10T16:15:55Z)
• Bidirectional Representations for Low Resource Spoken Language Understanding [39.208462511430554]
  We propose a representation model to encode speech in bidirectional rich encodings. The approach uses a masked language modelling objective to learn the representations. We show that the performance of the resulting encodings is better than comparable models on multiple datasets.
  arXiv  Detail & Related papers  (2022-11-24T17:05:16Z)
• Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS) [3.57486761615991]
  unified representations consistently achieves better cross-lingual synthesis with respect to both naturalness and accent. Separate representations tend to have an order of magnitude more tokens than unified ones, which may affect model capacity.
  arXiv  Detail & Related papers  (2022-07-04T16:14:57Z)
• A Brief Overview of Unsupervised Neural Speech Representation Learning [12.850357461259197]
  We review the development of unsupervised representation learning for speech over the last decade. We identify two primary model categories: self-supervised methods and probabilistic latent variable models.
  arXiv  Detail & Related papers  (2022-03-01T11:15:35Z)
• Interpreting Language Models with Contrastive Explanations [99.7035899290924]
  Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
  arXiv  Detail & Related papers  (2022-02-21T18:32:24Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training [135.12061144759517]
  We present an information-theoretic framework that formulates cross-lingual language model pre-training. We propose a new pre-training task based on contrastive learning. By leveraging both monolingual and parallel corpora, we jointly train the pretext to improve the cross-lingual transferability of pre-trained models.
  arXiv  Detail & Related papers  (2020-07-15T16:58:01Z)
• A Simple Joint Model for Improved Contextual Neural Lemmatization [60.802451210656805]
  We present a simple joint neural model for lemmatization and morphological tagging that achieves state-of-the-art results on 20 languages. Our paper describes the model in addition to training and decoding procedures.
  arXiv  Detail & Related papers  (2019-04-04T02:03:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.