Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

• URL: http://arxiv.org/abs/2603.15295v1
• Date: Mon, 16 Mar 2026 13:57:38 GMT
• Title: Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies
• Abstract summary: We present curated paradigm-based datasets for four languages, designed to probe systematic cross-sentence knowledge of verb alternations.<n>The datasets comprise thousands of the Blackbird Language Matrices (BLMs) problems.<n>We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the datasets.
• Score: 1.0857263744676489
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) have shown remarkable performance across various sentence-based linguistic phenomena, yet their ability to capture cross-sentence paradigmatic patterns, such as verb alternations, remains underexplored. In this work, we present curated paradigm-based datasets for four languages, designed to probe systematic cross-sentence knowledge of verb alternations (change-of-state and object-drop constructions in English, German and Italian, and Hebrew binyanim). The datasets comprise thousands of the Blackbird Language Matrices (BLMs) problems. The BLM task -- an RPM/ARC-like task devised specifically for language -- is a controlled linguistic puzzle where models must select the sentence that completes a pattern according to syntactic and semantic rules. We introduce three types of templates varying in complexity and apply linguistically-informed data augmentation strategies across synthetic and natural data. We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the datasets.

Related papers

• Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models [2.1390972559320653]
  This article describes a novel language task, the Blackbird Language Matrices (BLM) task, inspired by intelligence tests.<n>It illustrates the BLM datasets, their construction and benchmarking, and targeted experiments on chunking and systematicity.
  arXiv  Detail & Related papers  (2026-02-24T14:45:08Z)
• Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew [1.0857263744676489]
  We investigate how transformer models represent complex verb paradigms in Turkish and Modern Hebrew.<n>We show that for Turkish, both monolingual and multilingual models succeed, either when tokenization is atomic or when it breaks words into small subword units.<n>For Hebrew, instead, monolingual and multilingual models diverge.
  arXiv  Detail & Related papers  (2026-02-05T13:31:21Z)
• Improving Large Vision-Language Models' Understanding for Field Data [62.917026891829025]
  We introduce FieldLVLM, a framework designed to improve large vision-language models' understanding of field data.<n>FieldLVLM consists of two main components: a field-aware language generation strategy and a data-compressed multimodal model tuning.<n> Experimental results on newly proposed benchmark datasets demonstrate that FieldLVLM significantly outperforms existing methods in tasks involving scientific field data.
  arXiv  Detail & Related papers  (2025-07-24T11:28:53Z)
• The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs [45.08958917457921]
  Large language models (LLMs) still struggle across tasks outside of high-resource languages.<n>In this work, we investigate cross-lingual transfer to lower-resource languages where task-specific post-training data is scarce.
  arXiv  Detail & Related papers  (2025-05-23T20:28:31Z)
• New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaboration [49.180693704510006]
  Referring Expression (REC) is a cross-modal task that evaluates the interplay of language understanding, image comprehension, and language-to-image grounding.<n>It serves as an essential testing ground for Multimodal Large Language Models (MLLMs)
  arXiv  Detail & Related papers  (2025-02-27T13:58:44Z)
• Exploring Italian sentence embeddings properties through multi-tasking [1.4335183427838039]
  We study how sentence representations built using pre-trained language models encode specific syntactic and semantic information.<n>We use a two-level architecture to model separately a compression of the sentence embeddings into a representation that contains relevant information for a task, and a BLM task.<n>While we expected that the sentence structure -- in terms of sequence of phrases/chunks -- and chunk properties could be shared across tasks, performance and error analysis show that the clues for the different tasks are encoded in different manners in the sentence embeddings.
  arXiv  Detail & Related papers  (2024-09-10T16:22:18Z)
• Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement [1.4335183427838039]
  We take the approach of developing curated synthetic data on a large scale, with specific properties.<n>We use a new multiple-choice task and datasets, Blackbird Language Matrices, to focus on a specific grammatical structural phenomenon.<n>We show that despite having been trained on multilingual texts in a consistent manner, multilingual pretrained language models have language-specific differences.
  arXiv  Detail & Related papers  (2024-09-10T14:58:55Z)
• The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants [80.4837840962273]
  We present Belebele, a dataset spanning 122 language variants. This dataset enables the evaluation of text models in high-, medium-, and low-resource languages.
  arXiv  Detail & Related papers  (2023-08-31T17:43:08Z)
• CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models [77.45934004406283]
  We systematically study decompounding, the task of splitting compound words into their constituents. We introduce a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We introduce a novel methodology to train dedicated models for decompounding.
  arXiv  Detail & Related papers  (2023-05-23T16:32:27Z)
• Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval [80.43859162884353]
  We propose a multilingual multilingual language model called masked sentence model (MSM)<n>MSM consists of a sentence encoder to generate the sentence representations, and a document encoder applied to a sequence of sentence vectors from a document.<n>To train the model, we propose a masked sentence prediction task, which masks and predicts the sentence vector via a hierarchical contrastive loss with sampled negatives.
  arXiv  Detail & Related papers  (2023-02-03T09:54:27Z)
• Blackbird's language matrices (BLMs): a new benchmark to investigate disentangled generalisation in neural networks [2.5567566997688034]
  We illustrate Blackbird's language matrices (BLMs), a novel grammatical dataset developed to test a linguistic variant of Raven's progressive matrices. The dataset consists of 44800 sentences, generatively constructed to support investigations of current models' linguistic mastery of grammatical agreement rules. We show that this language task and the data that instantiate it provide a new challenging testbed to understand generalisation and abstraction.
  arXiv  Detail & Related papers  (2022-05-22T16:51:24Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.