XLM: A Python package for non-autoregressive language models

• URL: http://arxiv.org/abs/2512.17065v1
• Date: Thu, 18 Dec 2025 21:05:10 GMT
• Title: XLM: A Python package for non-autoregressive language models
• Authors: Dhruvesh Patel, Durga Prasad Maram, Sai Sreenivas Chintha, Benjamin Rozonoyer, Andrew McCallum,
• Abstract summary: In recent years, there has been a resurgence of interest in non-autoregressive text generation in the context of general language modeling.<n>We present the XLM python package, which is designed to make implementing small non-autoregressive language models faster.
• Score: 22.95539075474856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, there has been a resurgence of interest in non-autoregressive text generation in the context of general language modeling. Unlike the well-established autoregressive language modeling paradigm, which has a plethora of standard training and inference libraries, implementations of non-autoregressive language modeling have largely been bespoke making it difficult to perform systematic comparisons of different methods. Moreover, each non-autoregressive language model typically requires it own data collation, loss, and prediction logic, making it challenging to reuse common components. In this work, we present the XLM python package, which is designed to make implementing small non-autoregressive language models faster with a secondary goal of providing a suite of small pre-trained models (through a companion xlm-models package) that can be used by the research community. The code is available at https://github.com/dhruvdcoder/xlm-core.

Related papers

• Pretraining Language Models to Ponder in Continuous Space [50.52734567589996]
  We introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step.<n>We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations.
  arXiv  Detail & Related papers  (2025-05-27T03:47:33Z)
• Knowledge Fusion By Evolving Weights of Language Models [5.354527640064584]
  This paper examines the approach of integrating multiple models into a unified model. We propose a knowledge fusion method named Evolver, inspired by evolutionary algorithms.
  arXiv  Detail & Related papers  (2024-06-18T02:12:34Z)
• CyNetDiff -- A Python Library for Accelerated Implementation of Network Diffusion Models [0.9831489366502302]
  CyNetDiff is a Python library with components written in Cython to provide improved performance for these computationally intensive diffusion tasks. In many research tasks, these simulations are the most computationally intensive task, so it would be desirable to have a library for these with an interface to a high-level language.
  arXiv  Detail & Related papers  (2024-04-25T21:59:55Z)
• Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining [4.38070902806635]
  We set up a benchmark for languages Croatian, Serbian, Bosnian and Montenegrin. We show that comparable performance to dedicated from-scratch models can be obtained by additionally pretraining available multilingual models. We also show that neighboring languages, in our case Slovenian, can be included in the additional pretraining with little to no loss in the performance of the final model.
  arXiv  Detail & Related papers  (2024-04-08T11:55:44Z)
• FiLM: Fill-in Language Models for Any-Order Generation [71.42044325886194]
  Fill-in Language Model (FiLM) is a new language modeling approach that allows for flexible generation at any position without adhering to a specific generation order. During inference, FiLM can seamlessly insert missing phrases, sentences, or paragraphs. FiLM outperforms existing infilling methods that rely on left-to-right language models trained on rearranged text segments.
  arXiv  Detail & Related papers  (2023-10-15T19:37:39Z)
• Qwen Technical Report [132.54304067403922]
  We introduce Qwen, the first installment of our large language model series. Qwen is the base pretrained language models, and Qwen-Chat, the chat models finetuned with human alignment techniques. We have also developed coding-specialized models, Code-Qwen and Code-Qwen-Chat, as well as mathematics-focused models, Math-Qwen-Chat.
  arXiv  Detail & Related papers  (2023-09-28T17:07:49Z)
• Generate to Understand for Representation [3.5325087487696463]
  GUR is a pretraining framework that combines language modeling and contrastive learning objectives in a single training step. GUR achieves impressive results without any labeled training data, outperforming all other pretrained baselines as a retriever at the recall benchmark in a zero-shot setting.
  arXiv  Detail & Related papers  (2023-06-14T06:00:18Z)
• Multi-lingual Evaluation of Code Generation Models [82.7357812992118]
  We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages. We are able to assess the performance of code generation models in a multi-lingual fashion.
  arXiv  Detail & Related papers  (2022-10-26T17:17:06Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Fusing Sentence Embeddings Into LSTM-based Autoregressive Language Models [20.24851041248274]
  We present an LSTM-based autoregressive language model which uses prefix embeddings (from a pretrained masked language model) via fusion. We find that fusion helps reliably in lowering the perplexity (16.74 $rightarrow$ 15.80), which is even preserved after a transfer to a dataset from a different domain. We also evaluate the best-performing fusion model by correlating its next word surprisal estimates with human reading times.
  arXiv  Detail & Related papers  (2022-08-04T02:13:03Z)
• Twist Decoding: Diverse Generators Guide Each Other [116.20780037268801]
  We introduce Twist decoding, a simple and general inference algorithm that generates text while benefiting from diverse models. Our method does not assume the vocabulary, tokenization or even generation order is shared.
  arXiv  Detail & Related papers  (2022-05-19T01:27:53Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.