Generalizable Foundation Models for Calorimetry via Mixtures-of-Experts and Parameter Efficient Fine Tuning

• URL: http://arxiv.org/abs/2603.28804v1
• Date: Fri, 27 Mar 2026 23:52:32 GMT
• Title: Generalizable Foundation Models for Calorimetry via Mixtures-of-Experts and Parameter Efficient Fine Tuning
• Abstract summary: We introduce a generalizable foundation model for calorimetry built on next-token transformer backbones.<n>A pre-trained backbone is trained to generate electromagnetic showers across multiple materials.<n>New materials are incorporated through the addition and tuning of lightweight expert modules.
• Score: 0.005473229173811305
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern particle physics experiments face an increasing demand for high-fidelity detector simulation as luminosities rise and computational requirements approach the limits of available resources. Deep generative models have emerged as promising surrogates for traditional Monte Carlo simulation, with recent advances drawing inspiration from large language models (LLM) and next-token prediction paradigms. In this work, we introduce a generalizable foundation model for calorimetry built on next-token transformer backbones, designed to support modular adaptation across materials, particle species, and detector configurations. Our approach combines Mixture-of-Experts pre-training with parameter-efficient fine-tuning strategies to enable controlled, additive model expansion without catastrophic forgetting. A pre-trained backbone is trained to generate electromagnetic showers across multiple absorber materials, while new materials are incorporated through the addition and tuning of lightweight expert modules. Extensions to new particle types are achieved via parameter-efficient fine-tuning and modular vocabularies, preserving the integrity of the base model. This design enables efficient, incremental knowledge integration as new simulation datasets become available, a critical requirement in realistic detector-development workflows. In addition, we demonstrate that next-token calorimeter models are computationally competitive with standard generative approaches under established LLM optimization procedures. These results establish next-token architectures as a viable path toward extensible, physics-aware foundation models for calorimetry and future high-energy physics experiments.

Related papers

• GPT-Based Fast Simulation of CLAS12 Detector Hits via Conditional Autoregressive Generation [0.0]
  We present an application of a GPT-style autoregressive transformer as a fast surrogate model for the calorimeter inside the CLAS12 experiment.<n>We demonstrate that the model faithfully reproduces hit multiplicity, spatial distributions, energy deposits, and the energy-momentum response of the electromagnetic calorimeter.
  arXiv  Detail & Related papers  (2026-06-14T22:08:18Z)
• Fractional-order Modeling for Nonlinear Soft Actuators via Particle Swarm Optimization [49.34452620841728]
  This paper proposes an innovative modeling framework based on fractional-order differential equations (FODEs) to capture the dynamic behavior of soft materials.<n>The proposed approach effectively represents the complex deformation phenomena inherent in soft actuators.
  arXiv  Detail & Related papers  (2025-12-20T04:46:25Z)
• Fusion-Based Neural Generalization for Predicting Temperature Fields in Industrial PET Preform Heating [0.4337994560632144]
  We propose a novel deep learning framework for generalized temperature prediction.<n>Unlike traditional models that require extensive retraining for each material or design variation, our method introduces a data-efficient neural architecture.<n>Our approach reduces the need for large simulation datasets while achieving superior performance compared to models trained from scratch.
  arXiv  Detail & Related papers  (2025-10-06T21:38:37Z)
• Towards Machine Learning-based Model Predictive Control for HVAC Control in Multi-Context Buildings at Scale via Ensemble Learning [16.62153443064872]
  Building thermodynamics models predict real-time indoor temperature changes under potential HVAC control operations.<n>These models often require extensive data collection periods and rely heavily on expert knowledge, making the modeling process inefficient and limiting the reusability of the models.<n>This paper explores a model ensemble perspective that utilizes existing developed models as base models to serve a target building environment.
  arXiv  Detail & Related papers  (2025-05-05T08:09:36Z)
• Can Kans (re)discover predictive models for Direct-Drive Laser Fusion? [11.261403205522694]
  The domain of laser fusion presents a unique and challenging predictive modeling application landscape for machine learning methods. Data-driven approaches have been successful in the past for achieving desired generalization ability and model interpretation that aligns with physics expectations. In this work, we present the use of Kolmogorov-Arnold Networks (KANs) as an alternative to PIL for developing a new type of data-driven predictive model.
  arXiv  Detail & Related papers  (2024-09-13T13:48:06Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Koopman-Based Surrogate Modelling of Turbulent Rayleigh-Bénard Convection [4.248022697109535]
  We use a Koopman-inspired architecture called the Linear Recurrent Autoencoder Network (LRAN) for learning reduced-order dynamics in convection flows. A traditional fluid dynamics method, the Kernel Dynamic Mode Decomposition (KDMD) is used to compare the LRAN. We obtained more accurate predictions with the LRAN than with KDMD in the most turbulent setting.
  arXiv  Detail & Related papers  (2024-05-10T12:15:02Z)
• Improved particle-flow event reconstruction with scalable neural networks for current and future particle detectors [1.4609888393206634]
  We study scalable machine learning models for event reconstruction in electron-positron collisions based on a full detector simulation. We compare a graph neural network and kernel-based transformer and demonstrate that we can avoid operations while achieving realistic reconstruction. The best graph neural network model shows improvement in the jet transverse momentum resolution by up to 50% compared to the rule-based algorithm.
  arXiv  Detail & Related papers  (2023-09-13T08:16:15Z)
• Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture [68.13678918660872]
  We design a more capable parameter-sharing architecture based on matrix product operator (MPO) MPO decomposition can reorganize and factorize the information of a parameter matrix into two parts. Our architecture shares the central tensor across all layers for reducing the model size.
  arXiv  Detail & Related papers  (2023-03-27T02:34:09Z)
• Prediction of liquid fuel properties using machine learning models with Gaussian processes and probabilistic conditional generative learning [56.67751936864119]
  The present work aims to construct cheap-to-compute machine learning (ML) models to act as closure equations for predicting the physical properties of alternative fuels. Those models can be trained using the database from MD simulations and/or experimental measurements in a data-fusion-fidelity approach. The results show that ML models can predict accurately the fuel properties of a wide range of pressure and temperature conditions.
  arXiv  Detail & Related papers  (2021-10-18T14:43:50Z)
• MoEfication: Conditional Computation of Transformer Models for Efficient Inference [66.56994436947441]
  Transformer-based pre-trained language models can achieve superior performance on most NLP tasks due to large parameter capacity, but also lead to huge computation cost. We explore to accelerate large-model inference by conditional computation based on the sparse activation phenomenon. We propose to transform a large model into its mixture-of-experts (MoE) version with equal model size, namely MoEfication.
  arXiv  Detail & Related papers  (2021-10-05T02:14:38Z)
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration [130.89746032163106]
  We propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data. We show that the energy function and sampler can be trained efficiently via a new variational form of power iteration. We present an energy model guided fuzzer for software testing that achieves comparable performance to well engineered fuzzing engines like libfuzzer.
  arXiv  Detail & Related papers  (2020-11-10T19:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.