QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery

• URL: http://arxiv.org/abs/2602.17478v1
• Date: Thu, 19 Feb 2026 15:44:41 GMT
• Title: QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery
• Authors: Xuan-Bac Nguyen, Hoang-Quan Nguyen, Sankalp Pandey, Tim Faltermeier, Nicholas Borys, Hugh Churchill, Khoa Luu,
• Abstract summary: Characterizing two-dimensional quantum materials from optical microscopy images is challenging due to subtle layer-dependent contrast, limited labeled data, and significant variation across laboratories and imaging setups.<n>This work presents a new physics-aware multimodal framework that addresses these limitations from both the data and model perspectives.<n>We first present Synthia, a physics-based synthetic data generator that simulates realistic optical responses of quantum material flakes under thin-film interference.<n>We introduce QMat-Instruct, the first large-scale instruction dataset for quantum materials, comprising multimodal, physics-informed question-answer pairs designed to teach Multimodal Large Language Models (ML
• Score: 12.888415301529891
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Characterizing two-dimensional quantum materials from optical microscopy images is challenging due to the subtle layer-dependent contrast, limited labeled data, and significant variation across laboratories and imaging setups. Existing vision models struggle in this domain since they lack physical priors and cannot generalize to new materials or hardware conditions. This work presents a new physics-aware multimodal framework that addresses these limitations from both the data and model perspectives. We first present Synthia, a physics-based synthetic data generator that simulates realistic optical responses of quantum material flakes under thin-film interference. Synthia produces diverse and high-quality samples, helping reduce the dependence on expert manual annotation. We introduce QMat-Instruct, the first large-scale instruction dataset for quantum materials, comprising multimodal, physics-informed question-answer pairs designed to teach Multimodal Large Language Models (MLLMs) to understand the appearance and thickness of flakes. Then, we propose Physics-Aware Instruction Tuning (QuPAINT), a multimodal architecture that incorporates a Physics-Informed Attention module to fuse visual embeddings with optical priors, enabling more robust and discriminative flake representations. Finally, we establish QF-Bench, a comprehensive benchmark spanning multiple materials, substrates, and imaging settings, offering standardized protocols for fair and reproducible evaluation.

Related papers

• Exploring Physical Intelligence Emergence via Omni-Modal Architecture and Physical Data Engine [50.62040226184694]
  We present OmniFysics, a compact omni-modal model that unifies understanding across images, audio, video, and text.<n>To inject explicit physical knowledge, we build a physical data engine with two components.<n>Experiments show competitive performance on standard multimodal benchmarks and improved results on physics-oriented evaluations.
  arXiv  Detail & Related papers  (2026-02-05T14:04:51Z)
• PI-Light: Physics-Inspired Diffusion for Full-Image Relighting [26.42056487076843]
  We introduce Physics-Inspired diffusion for full-image reLight ($$-Light, or PI-Light), a two-stage framework that leverages physics-inspired diffusion models.<n>Our design incorporates (i) batch-aware attention, (ii) a physics-guided neural rendering module that enforces physically plausible light transport, and (iii) physics-inspired losses that regularize training dynamics toward a physically meaningful landscape.<n>Experiments demonstrate that $$-Light synthesizes specular highlights and diffuse reflections across a wide variety of materials, achieving superior generalization to real-world scenes compared with prior approaches.
  arXiv  Detail & Related papers  (2026-01-29T18:55:36Z)
• $\varphi$-Adapt: A Physics-Informed Adaptation Learning Approach to 2D Quantum Material Discovery [7.615935942148471]
  Characterizing quantum flakes is a critical step in quantum hardware engineering because the quality of these flakes directly influences qubit performance.<n>Computer vision methods for identifying two-dimensional quantum flakes have emerged, but they still face significant challenges in estimating flake thickness.<n>We introduce one of the first Physics-informed Adaptation Learning approaches to overcome these obstacles.
  arXiv  Detail & Related papers  (2025-07-07T16:40:35Z)
• PhysGaia: A Physics-Aware Dataset of Multi-Body Interactions for Dynamic Novel View Synthesis [62.283499219361595]
  PhysGaia is a physics-aware dataset specifically designed for Dynamic Novel View Synthesis (DyNVS)<n>Our dataset provides complex dynamic scenarios with rich interactions among multiple objects.<n>PhysGaia will significantly advance research in dynamic view synthesis, physics-based scene understanding, and deep learning models integrated with physical simulation.
  arXiv  Detail & Related papers  (2025-06-03T12:19:18Z)
• MAPS: Advancing Multi-Modal Reasoning in Expert-Level Physical Science [62.96434290874878]
  Current Multi-Modal Large Language Models (MLLM) have shown strong capabilities in general visual reasoning tasks.<n>We develop a new framework, named Multi-Modal Scientific Reasoning with Physics Perception and Simulation (MAPS) based on an MLLM.<n>MAPS decomposes expert-level multi-modal reasoning task into physical diagram understanding via a Physical Perception Model (PPM) and reasoning with physical knowledge via a simulator.
  arXiv  Detail & Related papers  (2025-01-18T13:54:00Z)
• IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations [64.07859467542664]
  Capturing geometric and material information from images remains a fundamental challenge in computer vision and graphics.<n>Traditional optimization-based methods often require hours of computational time to reconstruct geometry, material properties, and environmental lighting from dense multi-view inputs.<n>We introduce IDArb, a diffusion-based model designed to perform intrinsic decomposition on an arbitrary number of images under varying illuminations.
  arXiv  Detail & Related papers  (2024-12-16T18:52:56Z)
• OpenMaterial: A Large-scale Dataset of Complex Materials for 3D Reconstruction [55.052637670485716]
  We introduce OpenMaterial, a large-scale semi-synthetic dataset for material-aware 3D reconstruction.<n>It comprises 1,001 objects spanning 295 distinct materials, including conductors, dielectrics, plastics, and their roughened variants, captured under 714 diverse lighting conditions.<n>It provides multi-view images, 3D shape models, camera poses, depth maps, and object masks, establishing the first extensive benchmark for evaluating 3D reconstruction on challenging materials.
  arXiv  Detail & Related papers  (2024-06-13T07:46:17Z)
• Quantum-informed simulations for mechanics of materials: DFTB+MBD framework [40.83978401377059]
  We study how quantum effects can modify the mechanical properties of systems relevant to materials engineering. We provide an open-source repository containing all codes, datasets, and examples presented in this work.
  arXiv  Detail & Related papers  (2024-04-05T16:59:01Z)
• MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems [0.0]
  We report a flexible multi-modal mechanics language model, MeLM, applied to solve various nonlinear forward and inverse problems. The framework is applied to various examples including bio-inspired hierarchical honeycomb design and carbon nanotube mechanics.
  arXiv  Detail & Related papers  (2023-06-30T10:28:20Z)
• PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification [64.61198351207752]
  Existing approaches to system identification (estimating the physical parameters of an object) from videos assume known object geometries. In this work, we aim to identify parameters characterizing a physical system from a set of multi-view videos without any assumption on object geometry or topology. We propose "Physics Augmented Continuum Neural Radiance Fields" (PAC-NeRF), to estimate both the unknown geometry and physical parameters of highly dynamic objects from multi-view videos.
  arXiv  Detail & Related papers  (2023-03-09T18:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.