BigBang-Proton Technical Report: Next-Word-Prediction is Scientific Multitask Learner

• URL: http://arxiv.org/abs/2510.00129v1
• Date: Tue, 30 Sep 2025 18:09:18 GMT
• Title: BigBang-Proton Technical Report: Next-Word-Prediction is Scientific Multitask Learner
• Authors: Hengkui Wu, Liujiang Liu, Jihua He, Qihao Wang, Keke Zhao, Shuyang Hu, Renle Fu, Dahao Liang, Lingyu Zeng, Bruce Liu, Yuan Liu, Jin Zhan, Jiaqiang Niu, Xinglong Jia, Yaqin Hu, Wenjun Ji, Panpan Chi, Ken Chen, Hengyuan Wu, Yingsi Xin, Yongfeng Zhu, Yuexin Wang, Manqi Ruan, Ningtao Bian, Xiaohua Wu, Weipeng Xu,
• Abstract summary: BigBang-Proton is a unified sequence-based architecture for auto-regressive language modeling.<n>BigBang-Proton pretrained on cross-scale, cross-structure, cross-discipline real-world scientific tasks.
• Score: 8.599603915677365
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce BigBang-Proton, a unified sequence-based architecture for auto-regressive language modeling pretrained on cross-scale, cross-structure, cross-discipline real-world scientific tasks to construct a scientific multi-task learner. BigBang-Proton incorporates three fundamental innovations compared to mainstream general-purpose LLMs: Theory-Experiment Learning paradigm aligns large-scale numerical experimental data with theoretical text corpora; Binary Patch Encoding replaces byte pair encoding(BPE) tokenization; Monte Carlo Attention substitutes traditional transformer architectures. Through next-word-prediction pretraining on cross-discipline scientific datasets of real-world problems mixed with general textual corpus, followed by fine-tuning and inference on downstream tasks, BigBang-Proton demonstrates 100\% accuracy in up to 50-digit arithmetic addition operations, performance on par with leading specialized models in particle physics jet tagging, matching MAE of specialized models in inter-atomic potential simulation, performance comparable to traditional spatiotemporal models in water quality prediction, and benchmark-exceeding performance in genome modeling. These results prove that language-guided scientific computing can match or exceed the performance of task-specific scientific models while maintaining multitask learning capabilities. We further hypothesize to scale the pretraining to the universe scale as a fundamental step toward developing material world foundational model.

Related papers

• Beyond Language Modeling: An Exploration of Multimodal Pretraining [125.34714978184638]
  We provide empirical clarity through controlled, from-scratch pretraining experiments.<n>We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision.<n>We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language.
  arXiv  Detail & Related papers  (2026-03-03T18:58:00Z)
• ERNIE 5.0 Technical Report [244.36480708815316]
  ERNIE 5.0 is a unified autoregressive foundation model for unified multimodal understanding and generation across text, image, video, and audio.<n>To address practical challenges in large-scale deployment under diverse resource constraints, ERNIE 5.0 adopts a novel elastic training paradigm.<n>We show that ERNIE 5.0 achieves strong and balanced performance across multiple modalities.
  arXiv  Detail & Related papers  (2026-02-04T16:18:15Z)
• Innovator-VL: A Multimodal Large Language Model for Scientific Discovery [84.15264653078826]
  We present Innovator-VL, a scientific multimodal large language model designed to advance understanding and reasoning across diverse scientific domains.<n>We show that principled training design and transparent methodology can yield strong scientific intelligence with substantially reduced data requirements.
  arXiv  Detail & Related papers  (2026-01-27T08:12:18Z)
• Forecasting N-Body Dynamics: A Comparative Study of Neural Ordinary Differential Equations and Universal Differential Equations [4.285464959472458]
  The n body problem, fundamental to astrophysics, simulates the motion of n bodies acting under the effect of their own gravitational interactions.<n>Traditional machine learning models that are used for predicting and forecasting trajectories are often data intensive black box models.<n>Whereas Scientific Machine Learning directly embeds the known physical laws into the machine learning framework.
  arXiv  Detail & Related papers  (2025-12-12T11:20:27Z)
• A Mixture of Experts Gating Network for Enhanced Surrogate Modeling in External Aerodynamics [0.28647133890966997]
  Mixture of Experts (MoE) model combines predictions from three heterogeneous, state-of-the-art surrogate models.<n>The entire system is trained and validated on the DrivAerML dataset, a large-scale, public benchmark of high-fidelity CFD simulations for automotive aerodynamics.
  arXiv  Detail & Related papers  (2025-08-28T22:34:10Z)
• Foundation Model for Skeleton-Based Human Action Understanding [56.89025287217221]
  This paper presents a Unified Skeleton-based Dense Representation Learning framework.<n>USDRL consists of a Transformer-based Dense Spatio-Temporal (DSTE), Multi-Grained Feature Decorrelation (MG-FD), and Multi-Perspective Consistency Training (MPCT)
  arXiv  Detail & Related papers  (2025-08-18T02:42:16Z)
• HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling [52.58723853697152]
  We propose a Hybrid Architecture Distillation (HAD) approach for DNA sequence modeling.<n>We employ the NTv2-500M as the teacher model and devise a grouping masking strategy.<n>Compared to models with similar parameters, our model achieved excellent performance.
  arXiv  Detail & Related papers  (2025-05-27T07:57:35Z)
• Large EEG-U-Transformer for Time-Step Level Detection Without Pre-Training [1.3254304182988286]
  We propose a simple U-shaped model to efficiently learn representations by capturing both local and global features.<n>Compared to other window-level classification models, our method directly outputs predictions at the time-step level.<n>Our model won 1st place in the 2025 "seizure detection challenge" organized in the International Conference on Artificial Intelligence in Epilepsy and Other Neurological Disorders.
  arXiv  Detail & Related papers  (2025-04-01T01:33:42Z)
• UniGenX: a unified generative foundation model that couples sequence, structure and function to accelerate scientific design across proteins, molecules and materials [62.72989417755985]
  We present UniGenX, a unified generative model for function in natural systems.<n>UniGenX represents heterogeneous inputs as a mixed stream of symbolic and numeric tokens.<n>It achieves state-of-the-art or competitive performance for the function-aware generation across domains.
  arXiv  Detail & Related papers  (2025-03-09T16:43:07Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• A Large-Scale Evaluation of Speech Foundation Models [110.95827399522204]
  We establish the Speech processing Universal PERformance Benchmark (SUPERB) to study the effectiveness of the foundation model paradigm for speech. We propose a unified multi-tasking framework to address speech processing tasks in SUPERB using a frozen foundation model followed by task-specialized, lightweight prediction heads.
  arXiv  Detail & Related papers  (2024-04-15T00:03:16Z)
• OmniArch: Building Foundation Model For Scientific Computing [35.41293100957156]
  We present OmniArch, the first prototype aiming at solving multi-scale and multi-physics scientific computing problems with physical alignment.<n>As far as we know, we first conduct 1D-2D-3D united pre-training on the PDEBench, and it sets not only new performance benchmarks for 1D, 2D, and 3D PDEs but also demonstrates exceptional adaptability to new physics via in-context and zero-shot learning approaches.
  arXiv  Detail & Related papers  (2024-02-25T07:19:01Z)
• Multiple Physics Pretraining for Physical Surrogate Models [41.26924657687872]
  We introduce multiple physics pretraining (MPP), an autoregnostic task-atemporal pretraining approach for physical modeling.<n>In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems.<n>We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on both pretraining and downstream tasks.
  arXiv  Detail & Related papers  (2023-10-04T17:29:19Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Robust Graph Representation Learning via Predictive Coding [46.22695915912123]
  Predictive coding is a message-passing framework initially developed to model information processing in the brain. In this work, we build models that rely on the message-passing rule of predictive coding. We show that the proposed models are comparable to standard ones in terms of performance in both inductive and transductive tasks.
  arXiv  Detail & Related papers  (2022-12-09T03:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.