A.X K2 Technical Report

• URL: http://arxiv.org/abs/2608.30181v1
• Date: Mon, 31 Aug 2026 03:04:30 GMT
• Title: A.X K2 Technical Report
• Abstract summary: We introduce A.X K2, a language model trained from scratch as a high-performance foundation for emphagentic applications.<n>A.X K2 improves over A.X K1 across the board, by over 30 percentage points on some benchmarks.<n>A.X K2 performs competitively against strong open-weight baselines, matching or exceeding them on math and Korean-language benchmarks.
• Score: 16.892621349469206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce A.X K2, a 688B-parameter Mixture-of-Experts (MoE) language model trained from scratch as a high-performance foundation for \emph{agentic} applications. Trained on approximately 8.5T tokens---fewer than its predecessor, A.X K1---on a smaller but higher-quality mixture with substantially expanded agentic and software-engineering data, it nonetheless improves over A.X K1 across the board, by over 30 percentage points on some benchmarks, reflecting large gains in token efficiency. To support long contexts efficiently, we introduce Sparse Gated Attention (SGA), which combines sparse attention with gated attention, and adopt Gated Norm (GN) to stabilize large-scale training. SGA is trained natively at 128K through a \emph{sparse} indexer warmup that optimizes the indexer against its own sparse top-$k$ selection rather than the dense attention distribution, making adaptation markedly cheaper: each query reads only 2,048 positions, yet long-context quality is unchanged and A.X K2 scores 94.6 on RULER out to 256K. The outlier suppression of GN in turn keeps 4-bit NVFP4 serving within one point of FP8 accuracy. A simple yet effective Think-Fusion recipe further lets users switch between thinking and non-thinking modes within a single unified model. Extensive evaluations show that A.X K2 performs competitively against strong open-weight baselines, matching or exceeding them on math and Korean-language benchmarks.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Improved Large Language Diffusion Models [115.83169414681767]
  We present emphiLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention.<n>iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs.
  arXiv  Detail & Related papers  (2026-06-24T02:51:36Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• Extending Puzzle for Mixture-of-Experts Reasoning Models with Application to GPT-OSS Acceleration [17.929617807312635]
  Reasoning-focused LLMs improve answer quality by generating longer reasoning traces, but the additional tokens dramatically increase serving cost.<n>We extend and apply Puzzle, a post-training neural architecture search (NAS) framework, to produce gpt-oss-puzzle-88B.<n>Our approach combines heterogeneous MoE expert pruning, selective replacement of full-context attention with window attention, FP8 KV-cache quantization with calibrated scales, and post-training reinforcement learning to recover accuracy.
  arXiv  Detail & Related papers  (2026-02-12T13:36:19Z)
• Gated Sparse Attention: Combining Computational Efficiency with Training Stability for Long-Context Language Models [0.0]
  Gated Sparse Attention (GSA) is an architecture that realizes the benefits of both sparse and gated attention.<n>GSA incorporates a gated lightning indexer with sigmoid activations that produce bounded, interpretable selection scores.
  arXiv  Detail & Related papers  (2026-01-12T20:33:39Z)
• K-EXAONE Technical Report [76.23621600385238]
  K-EXAONE is a large-scale multilingual language model developed by LG AI Research.<n>It supports a 256K-token context window and covers six languages: Korean, English, Spanish, German, Japanese, and Vietnamese.<n>We evaluate K-EXAONE on a comprehensive benchmark suite spanning reasoning, agentic, general, Korean, and multilingual abilities.
  arXiv  Detail & Related papers  (2026-01-05T02:30:59Z)
• EdgeJury: Cross-Reviewed Small-Model Ensembles for Truthful Question Answering on Serverless Edge Inference [0.0]
  We present EdgeJury, a lightweight ensemble framework that improves truthfulness and robustness.<n>On TruthfulQA (MC1), EdgeJury achieves 76.2% accuracy.<n>On a 200-question adversarial EdgeCases set, EdgeJury yields +48.2% relative gains.
  arXiv  Detail & Related papers  (2025-12-29T14:48:40Z)
• Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2 [3.309291427648113]
  Huawei's openPangu-Embedded-1B and openPangu-Embedded-7B integrate three distinct Chain-of-Thought (CoT) reasoning paradigms.<n>These CoT modes introduce substantial memory and latency overheads, posing challenges for practical deployment on Ascend NPUs.<n>This paper addresses these computational constraints by leveraging low-bit quantization, which transforms FP16 computations into more efficient integer arithmetic.
  arXiv  Detail & Related papers  (2025-12-29T10:50:23Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Kimi K2: Open Agentic Intelligence [118.78600121345099]
  Kimi K2 is a large language model with 32 billion activated parameters and 1 trillion total parameters.<n>Based on MuonClip, K2 was pre-trained on 15.5 trillion tokens with zero loss spike.<n>Kimi K2 achieves state-of-the-art performance among open-source non-thinking models.
  arXiv  Detail & Related papers  (2025-07-28T05:35:43Z)
• QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models [57.04178959678024]
  We show that the majority of inference computations for large generative models can be performed with both weights and activations being cast to 4 bits. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit. We provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x.
  arXiv  Detail & Related papers  (2023-10-13T17:15:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.