OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research

• URL: http://arxiv.org/abs/2607.16669v1
• Date: Sat, 18 Jul 2026 06:55:18 GMT
• Title: OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research
• Abstract summary: OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models.<n>The package includes 27 presets across nine familiar model families and documentation that progresses from LM fundamentals to architecture research.
• Score: 1.1654568410166084
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models while keeping their machinery visible. In OLM, model code reads like the architecture: components are ordinary modules, while Block, Residual, Repeat, and Parallel describe how they are wired. The resulting model can move unchanged from a teaching notebook to a complete pretraining run or a research ablation. OLM connects this readable model layer to tokenizers, local and streaming datasets, optimization, mixed precision, callbacks, checkpoints, and hardware-aware CPU, single-GPU, and single-node multi-GPU execution. We demonstrate the full path by tracing GPT-2 from diagram to code, launching a FineWeb-Edu training script, replacing one attention component, and letting AutoTrainer configure the available machine. The package includes 27 presets across nine familiar model families and documentation that progresses from LM fundamentals to architecture research. Validation shows close agreement with independent reference implementations, 90.6% four-GPU weak-scaling efficiency for a 348M-parameter workload, compact architecture edits, and positive early usability results. OLM is MIT-licensed and available through PyPI, GitHub, and its documentation site.

Related papers

• BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion [52.26069987232466]
  We propose BLooP, a simple training-free decoding intervention that encourages large language models to generate bigrams from the source document.<n>BLooP operates through a hash table lookup at each decoding step, requiring no training, fine-tuning, or model modification.<n>Human evaluation shows that BLooP significantly improves faithfulness without reducing readability.
  arXiv  Detail & Related papers  (2026-03-12T01:14:19Z)
• Can LLMs Write CI? A Study on Automatic Generation of GitHub Actions Configurations [0.0]
  Continuous Integration services, such as GitHub Actions, require developers to write YAML-based configurations.<n>Despite the increasing use of Large Language Models (LLMs) to automate software engineering tasks, their ability to generate CI configurations remains underexplored.<n>This paper presents a preliminary study evaluating six LLMs for generating GitHub Actions configurations from natural language descriptions.
  arXiv  Detail & Related papers  (2025-07-23T03:18:04Z)
• Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
  We model the subgraph retrieval task as a conditional generation task handled by small language models. Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models. Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T15:22:36Z)
• TinyAgent: Function Calling at the Edge [32.174966522801746]
  We present an end-to-end framework for training and deploying task-specific small language model agents capable of function calling for driving agentic systems at the edge. As a driving application, we demonstrate a local Siri-like system for Apple's MacBook that can execute user commands through text or voice input.
  arXiv  Detail & Related papers  (2024-09-01T04:23:48Z)
• DistML.js: Installation-free Distributed Deep Learning Framework for Web Browsers [40.48978035180545]
  "DistML.js" is a library designed for training and inference of machine learning models within web browsers. We provide a comprehensive explanation of DistML.js's design, API, and implementation, alongside practical applications.
  arXiv  Detail & Related papers  (2024-07-01T07:13:14Z)
• KerasCV and KerasNLP: Vision and Language Power-Ups [9.395199188271254]
  KerasCV and KerasNLP are extensions of the Keras API for Computer Vision and Natural Language Processing. These domain packages are designed to enable fast experimentation, with a focus on ease-of-use and performance. The libraries are fully open-source (Apache 2.0 license) and available on GitHub.
  arXiv  Detail & Related papers  (2024-05-30T16:58:34Z)
• OpenELM: An Efficient Language Model Family with Open Training and Inference Framework [26.741510071520658]
  We release OpenELM, a state-of-the-art open language model. With a parameter budget of approximately one billion parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo.
  arXiv  Detail & Related papers  (2024-04-22T23:12:03Z)
• Language Models are Universal Embedders [45.8316643119292]
  In the large language model (LLM) revolution, embedding is a key component of various systems.<n>We propose strategies for constructing embedders and introduce a universal evaluation benchmark.<n> Experimental results show that our trained model is proficient at generating good embeddings across languages and tasks.
  arXiv  Detail & Related papers  (2023-10-12T11:25:46Z)
• Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants [65.47222691674074]
  Muffin framework employs pre-trained vision-language models to act as providers of visual signals. UniMM-Chat dataset explores the complementarities of datasets to generate 1.1M high-quality and diverse multimodal instructions.
  arXiv  Detail & Related papers  (2023-10-01T12:35:18Z)
• Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs [49.88461345825586]
  This paper proposes a new framework to enhance the fine-grained image understanding abilities of MLLMs. We present a new method for constructing the instruction tuning dataset at a low cost by leveraging annotations in existing datasets. We show that our model exhibits a 5.2% accuracy improvement over Qwen-VL and surpasses the accuracy of Kosmos-2 by 24.7%.
  arXiv  Detail & Related papers  (2023-10-01T05:53:15Z)
• Prompt2Model: Generating Deployable Models from Natural Language Instructions [74.19816829003729]
  Large language models (LLMs) enable system builders to create competent NLP systems through prompting. In other ways, LLMs are a step backward from traditional special-purpose NLP models. We propose Prompt2Model, a general-purpose method that takes a natural language task description like the prompts provided to LLMs.
  arXiv  Detail & Related papers  (2023-08-23T17:28:21Z)
• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality [95.76661165594884]
  mPLUG-Owl is a training paradigm that equips large language models (LLMs) with multi-modal abilities. The training paradigm involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM. Experimental results show that our model outperforms existing multi-modal models.
  arXiv  Detail & Related papers  (2023-04-27T13:27:01Z)
• Paraphrastic Representations at Scale [134.41025103489224]
  We release trained models for English, Arabic, German, French, Spanish, Russian, Turkish, and Chinese languages. We train these models on large amounts of data, achieving significantly improved performance from the original papers.
  arXiv  Detail & Related papers  (2021-04-30T16:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.