Instella: Fully Open Language Models with Stellar Performance

• URL: http://arxiv.org/abs/2511.10628v2
• Date: Fri, 14 Nov 2025 02:08:46 GMT
• Title: Instella: Fully Open Language Models with Stellar Performance
• Authors: Jiang Liu, Jialian Wu, Xiaodong Yu, Yusheng Su, Prakamya Mishra, Gowtham Ramesh, Sudhanshu Ranjan, Chaitanya Manem, Ximeng Sun, Ze Wang, Pratik Prabhanjan Brahma, Zicheng Liu, Emad Barsoum,
• Abstract summary: Instella is a family of fully open three billion parameter language models.<n>Powered by Instinct MI300X, Instella is developed through large-scale pre-training, general-purpose instruction tuning, and alignment with human preferences.<n>We release two specialized variants: Instella-Long, capable of handling context lengths up to 128K tokens, and Instella-Math, a reasoning-focused model enhanced through supervised fine-tuning and reinforcement learning.
• Score: 24.52924932842208
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks, yet the majority of high-performing models remain closed-source or partially open, limiting transparency and reproducibility. In this work, we introduce Instella, a family of fully open three billion parameter language models trained entirely on openly available data and codebase. Powered by AMD Instinct MI300X GPUs, Instella is developed through large-scale pre-training, general-purpose instruction tuning, and alignment with human preferences. Despite using substantially fewer pre-training tokens than many contemporaries, Instella achieves state-of-the-art results among fully open models and is competitive with leading open-weight models of comparable size. We further release two specialized variants: Instella-Long, capable of handling context lengths up to 128K tokens, and Instella-Math, a reasoning-focused model enhanced through supervised fine-tuning and reinforcement learning on mathematical tasks. Together, these contributions establish Instella as a transparent, performant, and versatile alternative for the community, advancing the goal of open and reproducible language modeling research.

Related papers

• Pretraining Language Models to Ponder in Continuous Space [50.52734567589996]
  We introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step.<n>We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations.
  arXiv  Detail & Related papers  (2025-05-27T03:47:33Z)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging [124.91183814854126]
  Model merging seeks to combine multiple expert models into a single model.<n>We introduce a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation.<n>We find that model merging offers a promising way for building improved MLLMs without requiring training data.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• Ola: Pushing the Frontiers of Omni-Modal Language Model [88.72389428177942]
  We present Ola, an omni-modal language model that achieves competitive performance across image, video, and audio understanding.<n>Ola incorporates advanced visual understanding and audio recognition capabilities through several critical and effective improvements.<n>We aim to make Ola a fully open omni-modal understanding solution to advance future research in this emerging field.
  arXiv  Detail & Related papers  (2025-02-06T18:59:55Z)
• LLMic: Romanian Foundation Language Model [76.09455151754062]
  We present LLMic, a foundation language model designed specifically for the Romanian Language.<n>We show that fine-tuning LLMic for language translation after the initial pretraining phase outperforms existing solutions in English-to-Romanian translation tasks.
  arXiv  Detail & Related papers  (2025-01-13T22:14:45Z)
• Knowledge Fusion By Evolving Weights of Language Models [5.354527640064584]
  This paper examines the approach of integrating multiple models into a unified model. We propose a knowledge fusion method named Evolver, inspired by evolutionary algorithms.
  arXiv  Detail & Related papers  (2024-06-18T02:12:34Z)
• Arcee's MergeKit: A Toolkit for Merging Large Language Models [0.6374098147778188]
  MergeKit is a framework to efficiently merge models on any hardware.<n>To date, thousands of models have been merged by the open-source community.
  arXiv  Detail & Related papers  (2024-03-20T02:38:01Z)
• MindLLM: Pre-training Lightweight Large Language Model from Scratch, Evaluations and Domain Applications [46.337078949637345]
  We present MindLLM, a novel series of bilingual lightweight large language models, trained from scratch. A thorough account of experiences accrued during large model development is given, covering every step of the process. MindLLM consistently matches or surpasses the performance of other open-source larger models on some public benchmarks.
  arXiv  Detail & Related papers  (2023-10-24T12:22:34Z)
• Open, Closed, or Small Language Models for Text Classification? [10.186568241388331]
  We evaluate three classes of models using eight datasets across three distinct NLP tasks. Open-source models can rival their closed-source counterparts by fine-tuning. This study underscores the importance of model selection based on task requirements.
  arXiv  Detail & Related papers  (2023-08-19T18:58:32Z)
• UnIVAL: Unified Model for Image, Video, Audio and Language Tasks [105.77733287326308]
  UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into a single model. Our model is efficiently pretrained on many tasks, based on task balancing and multimodal curriculum learning. Thanks to the unified model, we propose a novel study on multimodal model merging via weight generalization.
  arXiv  Detail & Related papers  (2023-07-30T09:48:36Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)
• Language Models are General-Purpose Interfaces [109.45478241369655]
  We propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language) We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders.
  arXiv  Detail & Related papers  (2022-06-13T17:34:22Z)
• Multitask Prompted Training Enables Zero-Shot Task Generalization [70.12770442071657]
  We develop a system for mapping general natural language tasks into a human-readable prompted form. We fine-tune a pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models 16x its size.
  arXiv  Detail & Related papers  (2021-10-15T17:08:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.