Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models

• URL: http://arxiv.org/abs/2401.13537v3
• Date: Thu, 11 Jul 2024 11:55:25 GMT
• Title: Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
• Authors: Tobias Golling, Lukas Heinrich, Michael Kagan, Samuel Klein, Matthew Leigh, Margarita Osadchy, John Andrew Raine,
• Abstract summary: Masked particle modeling (MPM) is a self-supervised method for learning generic, transferable, and reusable representations on unordered sets of inputs. We study the efficacy of the method in samples of high energy jets at collider physics experiments.
• Score: 4.299997052226609
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose masked particle modeling (MPM) as a self-supervised method for learning generic, transferable, and reusable representations on unordered sets of inputs for use in high energy physics (HEP) scientific data. This work provides a novel scheme to perform masked modeling based pre-training to learn permutation invariant functions on sets. More generally, this work provides a step towards building large foundation models for HEP that can be generically pre-trained with self-supervised learning and later fine-tuned for a variety of down-stream tasks. In MPM, particles in a set are masked and the training objective is to recover their identity, as defined by a discretized token representation of a pre-trained vector quantized variational autoencoder. We study the efficacy of the method in samples of high energy jets at collider physics experiments, including studies on the impact of discretization, permutation invariance, and ordering. We also study the fine-tuning capability of the model, showing that it can be adapted to tasks such as supervised and weakly supervised jet classification, and that the model can transfer efficiently with small fine-tuning data sets to new classes and new data domains.

Related papers

• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)
• Generative Modeling of Molecular Dynamics Trajectories [12.255021091552441]
  We introduce generative modeling of molecular trajectories as a paradigm for learning flexible multi-task surrogate models of MD from data. We show such generative models can be adapted to diverse tasks such as forward simulation, transition path sampling, and trajectory upsampling.
  arXiv  Detail & Related papers  (2024-09-26T13:02:28Z)
• Is Tokenization Needed for Masked Particle Modelling? [8.79008927474707]
  Masked particle modeling (MPM) is a self-supervised learning scheme for constructing expressive representations of unordered sets. We improve MPM by addressing inefficiencies in the implementation and incorporating a more powerful decoder. We show that these new methods outperform the tokenized learning objective from the original MPM on a new test bed for foundation models for jets.
  arXiv  Detail & Related papers  (2024-09-19T09:12:29Z)
• Enhancing Generative Molecular Design via Uncertainty-guided Fine-tuning of Variational Autoencoders [2.0701439270461184]
  A critical challenge for pre-trained generative molecular design models is to fine-tune them to be better suited for downstream design tasks. In this work, we propose a novel approach for a generative uncertainty decoder (VAE)-based GMD model through performance feedback in an active setting.
  arXiv  Detail & Related papers  (2024-05-31T02:00:25Z)
• MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining [73.81862342673894]
  Foundation models have reshaped the landscape of Remote Sensing (RS) by enhancing various image interpretation tasks. transferring the pretrained models to downstream tasks may encounter task discrepancy due to their formulation of pretraining as image classification or object discrimination tasks. We conduct multi-task supervised pretraining on the SAMRS dataset, encompassing semantic segmentation, instance segmentation, and rotated object detection. Our models are finetuned on various RS downstream tasks, such as scene classification, horizontal and rotated object detection, semantic segmentation, and change detection.
  arXiv  Detail & Related papers  (2024-03-20T09:17:22Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Quantum Generative Modeling of Sequential Data with Trainable Token Embedding [0.0]
  A quantum-inspired generative model known as the Born machines have shown great advancements in learning classical and quantum data. We generalize the embedding method into trainable quantum measurement operators that can be simultaneously honed with MPS. Our study indicated that combined with trainable embedding, Born machines can exhibit better performance and learn deeper correlations from the dataset.
  arXiv  Detail & Related papers  (2023-11-08T22:56:37Z)
• Exploring Model Transferability through the Lens of Potential Energy [78.60851825944212]
  Transfer learning has become crucial in computer vision tasks due to the vast availability of pre-trained deep learning models. Existing methods for measuring the transferability of pre-trained models rely on statistical correlations between encoded static features and task labels. We present an insightful physics-inspired approach named PED to address these challenges.
  arXiv  Detail & Related papers  (2023-08-29T07:15:57Z)
• Masked Autoencoding for Scalable and Generalizable Decision Making [93.84855114717062]
  MaskDP is a simple and scalable self-supervised pretraining method for reinforcement learning and behavioral cloning. We find that a MaskDP model gains the capability of zero-shot transfer to new BC tasks, such as single and multiple goal reaching.
  arXiv  Detail & Related papers  (2022-11-23T07:04:41Z)
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling [86.9726984929758]
  We focus on the integration of incomplete physics models into deep generative models. We propose a VAE architecture in which a part of the latent space is grounded by physics. We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-25T20:28:52Z)
• Embedded-physics machine learning for coarse-graining and collective variable discovery without data [3.222802562733787]
  We present a novel learning framework that consistently embeds underlying physics. We propose a novel objective based on reverse Kullback-Leibler divergence that fully incorporates the available physics in the form of the atomistic force field. We demonstrate the algorithmic advances in terms of predictive ability and the physical meaning of the revealed CVs for a bimodal potential energy function and the alanine dipeptide.
  arXiv  Detail & Related papers  (2020-02-24T10:28:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.