Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement

• URL: http://arxiv.org/abs/2409.06567v1
• Date: Tue, 10 Sep 2024 14:58:55 GMT
• Title: Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement
• Authors: Vivi Nastase, Chunyang Jiang, Giuseppe Samo, Paola Merlo,
• Abstract summary: We take the approach of developing curated synthetic data on a large scale, with specific properties. We use a new multiple-choice task and datasets, Blackbird Language Matrices, to focus on a specific grammatical structural phenomenon. We show that despite having been trained on multilingual texts in a consistent manner, multilingual pretrained language models have language-specific differences.
• Score: 1.4335183427838039
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, our goal is to investigate to what degree multilingual pretrained language models capture cross-linguistically valid abstract linguistic representations. We take the approach of developing curated synthetic data on a large scale, with specific properties, and using them to study sentence representations built using pretrained language models. We use a new multiple-choice task and datasets, Blackbird Language Matrices (BLMs), to focus on a specific grammatical structural phenomenon -- subject-verb agreement across a variety of sentence structures -- in several languages. Finding a solution to this task requires a system detecting complex linguistic patterns and paradigms in text representations. Using a two-level architecture that solves the problem in two steps -- detect syntactic objects and their properties in individual sentences, and find patterns across an input sequence of sentences -- we show that despite having been trained on multilingual texts in a consistent manner, multilingual pretrained language models have language-specific differences, and syntactic structure is not shared, even across closely related languages.

Related papers

• Finding Structure in Language Models [3.882018118763685]
  This thesis is about whether language models possess a deep understanding of grammatical structure similar to that of humans. We will develop novel interpretability techniques that enhance our understanding of the complex nature of large-scale language models.
  arXiv  Detail & Related papers  (2024-11-25T14:37:24Z)
• Learning Phonotactics from Linguistic Informants [54.086544221761486]
  Our model iteratively selects or synthesizes a data-point according to one of a range of information-theoretic policies. We find that the information-theoretic policies that our model uses to select items to query the informant achieve sample efficiency comparable to, or greater than, fully supervised approaches.
  arXiv  Detail & Related papers  (2024-05-08T00:18:56Z)
• Multilingual Multi-Figurative Language Detection [14.799109368073548]
  figurative language understanding is highly understudied in a multilingual setting. We introduce multilingual multi-figurative language modelling, and provide a benchmark for sentence-level figurative language detection. We develop a framework for figurative language detection based on template-based prompt learning.
  arXiv  Detail & Related papers  (2023-05-31T18:52:41Z)
• Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages [3.716965622352967]
  We propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks.
  arXiv  Detail & Related papers  (2023-05-26T18:06:49Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Are Multilingual Models Effective in Code-Switching? [57.78477547424949]
  We study the effectiveness of multilingual language models to understand their capability and adaptability to the mixed-language setting. Our findings suggest that pre-trained multilingual models do not necessarily guarantee high-quality representations on code-switching.
  arXiv  Detail & Related papers  (2021-03-24T16:20:02Z)
• Multilingual AMR-to-Text Generation [22.842874899794996]
  We create multilingual AMR-to-text models that generate in twenty one different languages. For eighteen languages, based on automatic metrics, our multilingual models surpass baselines that generate into a single language. We analyse the ability of our multilingual models to accurately capture morphology and word order using human evaluation, and find that native speakers judge our generations to be fluent.
  arXiv  Detail & Related papers  (2020-11-10T22:47:14Z)
• Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations [83.27475281544868]
  We use singular vector canonical correlation analysis to study what kind of information is induced from each source. We observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy.
  arXiv  Detail & Related papers  (2020-04-30T16:25:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.