TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI

• URL: http://arxiv.org/abs/2608.30567v1
• Date: Mon, 31 Aug 2026 10:42:35 GMT
• Title: TuringLLM: Efficiently Scaling Foundation Models Toward Physical AI
• Abstract summary: We present Turing-20B-A2B, a Mixture-of-Experts language model that activates approximately 2B parameters per token.<n>The model adopts Quantile Routing in a dynamic top-k configuration, enabling token-adaptive expert allocation.<n>The model is pretrained with a progressive three-stage curriculum and extended to a native context length of 128K through continued pretraining.
• Score: 43.58694189855896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Turing-20B-A2B, a 20B-parameter Mixture-of-Experts language model that activates approximately 2B parameters per token, designed for long-context and latency-sensitive physical AI applications. The model adopts Quantile Routing in a dynamic top-k configuration, enabling token-adaptive expert allocation while maintaining balanced expert utilization and a controlled average compute budget. During deployment, we further apply capacity-constrained routing to prompt prefill for more regular and efficient expert execution, while retaining dropless routing during pretraining. Turing-20B-A2B also employs a hybrid attention architecture that combines Lightning Attention with a small number of full-attention layers for efficient long-context modeling. The model is pretrained with a progressive three-stage curriculum and extended to a native context length of 128K through continued pretraining, with further inference-time extension to 512K using YaRN. Despite its compact active-parameter budget, Turing-20B-A2B achieves, at the base-model stage, overall general capability exceeding Qwen3-8B Base and approaching Qwen3.5-9B Base, while maintaining strong long-context performance and favorable prefill-latency scaling. These results demonstrate an effective balance among model capability, long-context scalability, and practical inference efficiency.

Related papers

• 1BT: One-Block Transformer for EEG-Based Cognitive Workload Assessment [6.076106785472718]
  This paper introduces 1BT, a One-Block Transformer for compact and efficient EEG-based cognitive workload assessment.<n>A controlled study involving 11 participants performing three cognitively diverse tasks was conducted.<n>The model achieves high workload classification performance with under 0.5 million parameters and 0.02 GFLOPs.
  arXiv  Detail & Related papers  (2026-04-21T04:32:59Z)
• LongCat-Flash-Thinking-2601 Technical Report [134.89732115690705]
  LongCat-Flash-Thinking-2601 is an open-source Mixture-of-Experts (MoE) reasoning model with superior agentic reasoning capability.<n>LongCat-Flash-Thinking-2601 achieves state-of-the-art performance among open-source models on a wide range of agentic benchmarks.
  arXiv  Detail & Related papers  (2026-01-23T13:20:09Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• LongCat-Flash Technical Report [165.64670448930875]
  LongCat-Flash is a 560-billion- parameter Mixture-of-Experts (MoE) language model.<n>It is designed for both computational efficiency and advanced agentic capabilities.<n>We complete model training on more than 20 trillion tokens within 30 days, while achieving over 100 tokens per second (TPS) for inference at a cost of $0.70 per million output tokens.
  arXiv  Detail & Related papers  (2025-09-01T10:05:45Z)
• KAT-V1: Kwai-AutoThink Technical Report [50.84483585850113]
  We present Kwaipilot-AutoThink (KAT), an open-source 40B large language model developed to address the overthinking problem in reasoning-intensive tasks.<n>KAT dynamically switches between reasoning and non-reasoning modes based on task complexity.<n>We also propose Step-SRPO, a reinforcement learning algorithm that incorporates intermediate supervision into the GRPO framework.
  arXiv  Detail & Related papers  (2025-07-11T04:07:10Z)
• LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders [23.70714095931094]
  Long-sequence optimized traNsformer for GPU-Efficient Recommenders.<n>Longer consistently outperforms strong baselines in offline metrics and online A/B testing.
  arXiv  Detail & Related papers  (2025-05-07T13:54:26Z)
• FFN Fusion: Rethinking Sequential Computation in Large Language Models [16.8637819797503]
  We introduce FFN Fusion, an architectural optimization technique that reduces sequential computation in large language models.<n>We develop a principled methodology for identifying and fusing such sequences, transforming them into parallel operations.<n>Applying these techniques to Llama-3.1-405B-Instruct, we create an efficient and soon-to-be publicly available model that achieves a 1.71X speedup in inference latency and 35X lower per-token cost.
  arXiv  Detail & Related papers  (2025-03-24T17:20:35Z)
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models [152.29364079385635]
  As pre-trained models grow bigger, the fine-tuning process can be time-consuming and computationally expensive. We propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning and (ii) resource-efficient inference.
  arXiv  Detail & Related papers  (2021-10-30T03:29:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.