EngGPT2: Sovereign, Efficient and Open Intelligence

• URL: http://arxiv.org/abs/2603.16430v2
• Date: Wed, 18 Mar 2026 12:11:44 GMT
• Title: EngGPT2: Sovereign, Efficient and Open Intelligence
• Abstract summary: EngGPT2 is built to be a Sovereign, Efficient and Open model.<n>It's trained on 2.5 trillion tokens - less than Qwen3's 36T or Llama3's 15T.<n>It delivers performance on key benchmarks, including MMLU-Pro, GSM8K, IFEval and HumanEval.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: EngGPT2-16B-A3B is the latest iteration of Engineering Group's Italian LLM and it's built to be a Sovereign, Efficient and Open model. EngGPT2 is trained on 2.5 trillion tokens - less than Qwen3's 36T or Llama3's 15T - and delivers performance on key benchmarks, including MMLU-Pro, GSM8K, IFEval and HumanEval, comparable to dense models in the 8B-16B range, while requiring one-fifth to half of the inference power, and between one-tenth to one-sixth of the training data and consequent needed training power. Designed as a trained-from-scratch Mixture-of-Experts (MoE) architecture, EngGPT2 features 16 billion parameters with 3 billion active per inference, with expert sizes positioned between those used in GPT-OSS and Qwen3. Approximately 25% of its training corpus consists of Italian-language data, to deliver strong capabilities for European and Italian NLP tasks among models of similar scale. This efficiency aims to position EngGPT2 as a key contributor to the growing portfolio of open-weight European models, combining performance and efficiency with full alignment to the EU AI Act. EngGPT2 is also a single model capable of multiple reasoning modes: non-reasoning, reasoning in Italian or English, and turbo-reasoning (a concise, bullet-point style reasoning available in both languages designed for real-time reasoning use cases). EngGPT2 aims to set a new standard for resource-conscious, high-performance LLMs tailored to European and Italian contexts.

Related papers

• STEP3-VL-10B Technical Report [115.89015065130127]
  STEP3-VL-10B is a lightweight foundation model designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence.<n>We implement Parallel Coordinated Reasoning (PaCoRe) to scale test-time compute, allocating resources to scalable perceptual reasoning.<n>It records 92.2% on MMBench and 80.11% on MMMU, while excelling in complex reasoning with 94.43% on AIME2025 and 75.95% on MathVision.
  arXiv  Detail & Related papers  (2026-01-14T17:58:24Z)
• Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM [30.381516759139203]
  We present Gamayun, a multilingual language model trained entirely from scratch on 2.5T tokens.<n>Our model supports 12 languages, with special focus on Russian.<n>It matches or exceeds Qwen3 (36T tokens) on most tasks outside advanced STEM.
  arXiv  Detail & Related papers  (2025-12-25T08:52:23Z)
• MiniLingua: A Small Open-Source LLM for European Languages [47.78284815754002]
  MiniLingua is a multilingual open-source LLM of one billion parameters trained from scratch for 13 European languages.<n>We release model weights, tokenizer and source code used for data processing and model training.
  arXiv  Detail & Related papers  (2025-12-15T13:12:42Z)
• LFM2 Technical Report [87.58431408281973]
  We present LFM2, a family of Liquid Foundation Models designed for efficient on-device deployment and strong task capabilities.<n>The LFM2 family covers 350M-8.3B parameters, including dense models (350M, 700M, 1.2B, 2.6B) and a mixture-of-experts variant (8.3B total, 1.5B active)<n>We build multimodal and retrieval variants: LFM2-VL for vision-latency tasks, LFM2-Audio for speech, and LFM2-ColBERT for retrieval.
  arXiv  Detail & Related papers  (2025-11-28T17:56:35Z)
• MobileLLM-Pro Technical Report [28.511762884727883]
  MobileLLM-Pro is a 1-billion- parameter language model optimized for on-device deployment.<n>It significantly outperforms Gemma 3-1B and Llama 3.2-1B on 11 standard benchmarks.<n>It supports context windows of up to 128,000 tokens and shows only minor performance regressions at 4-bit quantization.
  arXiv  Detail & Related papers  (2025-11-10T05:28:31Z)
• Empowering Smaller Models: Tuning LLaMA and Gemma with Chain-of-Thought for Ukrainian Exam Tasks [0.0]
  Small or compact models, though more efficient, often lack sufficient support for underrepresented languages.<n>This work explores the potential of parameter-efficient fine-tuning of compact open-weight language models to handle reasoning-intensive tasks.<n> tuning method with joint task topic and step-by-step solution generation outperforms standard chain-of-thought tuning in matching tasks.
  arXiv  Detail & Related papers  (2025-03-18T07:44:49Z)
• GEB-1.3B: Open Lightweight Large Language Model [12.083014082506281]
  We introduce GEB-1.3B, a lightweight large language model (LLMs) trained on 550 billion tokens in both Chinese and English languages. We employ novel training techniques, including ROPE, Group-Query-Attention, and FlashAttention-2, to accelerate training while maintaining model performance. GEB-1.3B exhibits outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU, outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B. The release of GEB-1.3B as an open-source model marks a significant contribution to the development
  arXiv  Detail & Related papers  (2024-06-14T10:15:49Z)
• YAYI 2: Multilingual Open-Source Large Language Models [53.92832054643197]
  We propose YAYI 2, including both base and chat models, with 30 billion parameters. YAYI 2 is pre-trained from scratch on a multilingual corpus which contains 2.65 trillion tokens filtered by our pre-training data processing pipeline. The base model is aligned with human values through supervised fine-tuning with millions of instructions and reinforcement learning from human feedback.
  arXiv  Detail & Related papers  (2023-12-22T17:34:47Z)
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts [84.33607245023049]
  We propose and develop a family of language models named GLaM (Generalist Language Model) GLaM uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.
  arXiv  Detail & Related papers  (2021-12-13T18:58:19Z)
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models [152.29364079385635]
  As pre-trained models grow bigger, the fine-tuning process can be time-consuming and computationally expensive. We propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning and (ii) resource-efficient inference.
  arXiv  Detail & Related papers  (2021-10-30T03:29:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.