Mimir: Large-scale Multilingual Concept Modeling

• URL: http://arxiv.org/abs/2605.25263v1
• Date: Sun, 24 May 2026 21:26:47 GMT
• Title: Mimir: Large-scale Multilingual Concept Modeling
• Abstract summary: We introduce Mimir, a 1.6B Large Concept Model trained for multilingual concept understanding and generation.<n>We leverage a large-scale multilingual pre-training corpus spanning 46 languages and a large-scale multi-turn and multilingual instruction-tuning dataset covering a total of 35 languages.
• Score: 1.9881456274482427
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Current language modeling approaches are built around tokens. Text corpora are split into tokens, and models are trained by performing computations on these tokens, such as predicting the next token given the preceding ones as context. This paradigm has become the standard in modern language modeling, especially given the outstanding performance obtained by token-based architectures. However, recent works have not only begun to question how language models process and understand meaning from tokens, but also to question whether using higher levels of granularity could advance the research field. This led to the idea of Concept Modeling, that is, to directly train models for next-concept prediction rather than next-token prediction. The goal is to change the input from tokens to concepts, forcing the underlying language model to shift its granularity from fine-grained tokens to broad concepts. In this work, we introduce Mimir, a 1.6B Large Concept Model trained for multilingual concept understanding and generation. We leverage a large-scale multilingual pre-training corpus (38,883,987,240 sentences) spanning 46 languages and a large-scale multi-turn and multilingual instruction-tuning dataset (66,816,428 sentences) covering a total of 35 languages. We extensively evaluate model performance against a language model with a comparable number of parameters.

Related papers

• Pretraining Finnish ModernBERTs [3.000523452333836]
  This paper reports on pretraining ModernBERT encoder models in six different sizes, ranging from 51M to 475M parameters, with a focus on limited multilingualism, emphasizing languages relevant to Finland.<n>Our models are competitive with, or superior to, existing multilingual models. They outperform monolingual models on tasks that require a context longer than 512 tokens.
  arXiv  Detail & Related papers  (2025-11-12T11:21:05Z)
• A Variational Framework for Improving Naturalness in Generative Spoken Language Models [52.673912922590866]
  We propose an end-to-end variational approach that automatically learns to encode continuous speech attributes to enhance semantic tokens.<n>Our approach eliminates the need for manual extraction and selection of paralinguistic features.<n>It produces preferred speech continuations according to human raters.
  arXiv  Detail & Related papers  (2025-06-17T17:58:17Z)
• From Bytes to Ideas: Language Modeling with Autoregressive U-Nets [49.16552366851748]
  Tokenization imposes a fixed granularity on the input text.<n>We introduce an autoregressive U-Net that learns to embed its own tokens as it trains.
  arXiv  Detail & Related papers  (2025-06-17T17:55:11Z)
• xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillation [2.9998889086656586]
  We propose an adaptation methodology for Large Vision-Language Models trained on English language data to improve their performance.<n>We introduce a benchmark to evaluate the effectiveness of multilingual and multimodal embedding models.
  arXiv  Detail & Related papers  (2025-03-12T12:04:05Z)
• Large Concept Models: Language Modeling in a Sentence Representation Space [62.73366944266477]
  We present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a concept.<n> Concepts are language- and modality-agnostic and represent a higher level idea or action in a flow.<n>We show that our model exhibits impressive zero-shot generalization performance to many languages.
  arXiv  Detail & Related papers  (2024-12-11T23:36:20Z)
• Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction [23.45902601618188]
  Language models have demonstrated impressive ability in context understanding and generative performance. We propose LMTraj (Language-based Multimodal Trajectory predictor), which recasts the trajectory prediction task into a sort of question-answering problem. We show that the language-based model can be a powerful pedestrian trajectory predictor, and outperforms existing numerical-based predictor methods.
  arXiv  Detail & Related papers  (2024-03-27T11:06:44Z)
• Pre-trained Language Models Do Not Help Auto-regressive Text-to-Image Generation [82.5217996570387]
  We adapt a pre-trained language model for auto-regressive text-to-image generation. We find that pre-trained language models offer limited help.
  arXiv  Detail & Related papers  (2023-11-27T07:19:26Z)
• Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks [12.7259425362286]
  We investigate how multilingual models might leverage key-value memories. For autoregressive models trained on two or more languages, do all neurons (across layers) respond equally to all languages? Our findings reveal that the layers closest to the network's input or output tend to exhibit more language-specific behaviour compared to the layers in the middle.
  arXiv  Detail & Related papers  (2023-10-24T06:45:00Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• Multi-Sense Language Modelling [19.396806939258806]
  We propose a language model which not only predicts the next word, but also its sense in context. This higher prediction granularity may be useful for end tasks such as assistive writing. For sense prediction, we utilise a Graph Attention Network, which encodes definitions and example uses of word senses.
  arXiv  Detail & Related papers  (2020-12-10T16:06:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.