Llamion Technical Report

• URL: http://arxiv.org/abs/2605.25676v1
• Date: Mon, 25 May 2026 10:27:07 GMT
• Title: Llamion Technical Report
• Abstract summary: We transform Orion-14B into the standardized Llama-family architecture.<n>Llamion recovers Orion's behaviour on H6, MT-Bench, and KoMMLU with only 123M tokens on a single A100 in four days.
• Score: 17.983567146196254
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We release Llamion, a family of 14B-parameter open-weight language models obtained by transforming Orion-14B into the standardized Llama-family architecture. The transformation is performed by Efficient Knowledge Preservation for Transformation (KEPT), a recipe that combines (i) Normal Parameter Mapping (NPM) for unchanged modules, (ii) Optimized Parameter Mapping (OPM), a training-free LayerNorm-to-RMSNorm initialization we prove optimal under the near-zero-mean activation regime induced by weight decay, and (iii) Cross-architecture Knowledge Distillation (XKD), an equal-size frozen-teacher distillation that aligns the converted model's outputs with the source model's on any reasonable input distribution. Llamion recovers Orion's behaviour on H6, MT-Bench, and KoMMLU with only ~123M tokens on a single A100 in four days; Llamion-Base reaches 66.87% on KoMMLU, exceeding the next-best entry of the Open Ko LLM Leaderboard by >7.0 absolute points at submission time. Capabilities entirely absent from the transfer corpus (Python programming and 200K-token context handling) survive the architectural transition intact. We release three checkpoints (Base, Chat, LongChat) that load with trust_remote_code=False in the Hugging Face Transformers library.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs [13.903045659324187]
  We present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs.<n>For pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples.<n>We show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours.
  arXiv  Detail & Related papers  (2026-06-25T06:24:02Z)
• A general tensor-structured compression scheme for efficient large language models [8.85443988406591]
  We present a general tensor-structured compression scheme that replaces targeted dense linear layers with executable mixtures of tensor operators.<n>MixT is potentially applicable across Transformer-based large language models (LLMs) and other dense neural mappings.<n>At the LLaMA2-7B transition boundary, MixT reduces full-model parameters by 47.5%, inference FLOPs by 37.1%, training FLOPs by 52.1% and peak inference memory by 60.4%.
  arXiv  Detail & Related papers  (2026-05-25T02:00:41Z)
• Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs [48.83701310501069]
  Large language models (LLMs) show strong potential for neural architecture generation.<n>Existing approaches produce complete model implementations from scratch.<n>We propose Delta-Code Generation, where fine-tuned LLMs generate compact unified diffs.
  arXiv  Detail & Related papers  (2026-05-06T13:32:05Z)
• THEIA: Learning Complete Kleene Three-Valued Logic in a Pure-Neural Modular Architecture [0.0]
  THEIA is a 2.75M modular neural architecture that learns the complete Kleene three-valued logic (K3) truth table from task data without external symbolic inference or hand-encoded K3 gate primitives.<n> Transformer baselines also reach >99% on all 39 rules, and flats match THEIA on Phase-1 accuracy within 0.04pp.
  arXiv  Detail & Related papers  (2026-04-13T10:44:15Z)
• Data-Free Layer-Adaptive Merging via Fisher Information for Long-to-Short Reasoning LLMs [3.18088418556364]
  We provide the first theoretical justification for layer-adaptive merging.<n>We propose textbfFIM-Merging, which computes diagonal FIM using only random token inputs.<n>Our framework also provides a unified theoretical explanation for why existing layer-adaptive methods such as ACM empirically outperform uniform merging.
  arXiv  Detail & Related papers  (2026-03-23T08:45:37Z)
• JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation [46.64215658042213]
  We introduce Joint-Token (JTok) and Mixture of Joint-Token (JTok-M), which augment Transformer layers with modulation vectors retrieved from auxiliary embedding tables.<n>These vectors modulate the backbone via lightweight, element-wise operations, incurring negligible FLOPs overhead.<n>Our approach consistently reduces validation loss and significantly improves downstream task performance.
  arXiv  Detail & Related papers  (2026-01-31T16:15:18Z)
• From Memorization to Creativity: LLM as a Designer of Novel Neural-Architectures [48.83701310501069]
  Large language models (LLMs) excel in program synthesis, yet their ability to autonomously navigate neural architecture design-balancing reliability, performance, and structural novelty--remains underexplored.<n>We address this by placing a code-oriented LLM within a closed-loop synthesis framework, analyzing its evolution over 22 supervised fine-tuning cycles.
  arXiv  Detail & Related papers  (2026-01-06T13:20:28Z)
• TransMLA: Multi-Head Latent Attention Is All You Need [34.38934956358534]
  TransMLA is a framework that seamlessly converts GQA-based models to MLA-based models.<n>By compressing 93% of the KV cache in LLaMA-2-7B, TransMLA achieves a 10.6x inference speedup at an 8K context length.
  arXiv  Detail & Related papers  (2025-02-11T18:20:18Z)
• DataComp-LM: In search of the next generation of training sets for language models [200.5293181577585]
  DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments with the goal of improving language models.<n>We provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations.<n>Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters.
  arXiv  Detail & Related papers  (2024-06-17T17:42:57Z)
• SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation [53.675725490807615]
  We introduce SDPose, a new self-distillation method for improving the performance of small transformer-based models. SDPose-T obtains 69.7% mAP with 4.4M parameters and 1.8 GFLOPs, while SDPose-S-V2 obtains 73.5% mAP on the MSCOCO validation dataset.
  arXiv  Detail & Related papers  (2024-04-04T15:23:14Z)
• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition [99.349598600887]
  Conformer is the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture. We propose the Squeezeformer model, which consistently outperforms the state-of-the-art ASR models under the same training schemes.
  arXiv  Detail & Related papers  (2022-06-02T06:06:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.