AstroCLIP: A Cross-Modal Foundation Model for Galaxies

• URL: http://arxiv.org/abs/2310.03024v2
• Date: Fri, 14 Jun 2024 17:19:58 GMT
• Title: AstroCLIP: A Cross-Modal Foundation Model for Galaxies
• Authors: Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Ruben Ohana, Mariel Pettee, Bruno Regaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, Shirley Ho,
• Abstract summary: AstroCLIP embeds galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We find remarkable performance on all downstream tasks, even relative to supervised baselines. Our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.
• Score: 40.43521617393482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.

Related papers

• AstroSpy: On detecting Fake Images in Astronomy via Joint Image-Spectral Representations [31.75799061059914]
  The prevalence of AI-generated imagery has raised concerns about the authenticity of astronomical images. We present AstroSpy, a hybrid model that integrates both spectral and image features to distinguish real from synthetic astronomical images.
  arXiv  Detail & Related papers  (2024-07-09T12:49:44Z)
• Galaxy spectroscopy without spectra: Galaxy properties from photometric images with conditional diffusion models [3.556281115019309]
  We develop a generative AI method capable of predicting optical galaxy spectra from photometric broad-band images alone. This work is the first attempt in the literature to infer velocity dispersion from photometric images. We can predict the presence of an active galactic nucleus up to an accuracy of 82%.
  arXiv  Detail & Related papers  (2024-06-26T08:49:51Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)
• SSG2: A new modelling paradigm for semantic segmentation [0.0]
  State-of-the-art models in semantic segmentation operate on single, static images, generating corresponding segmentation masks. Inspired by work on semantic change detection, we introduce a methodology that leverages a sequence of observables generated for each static input image. By adding this "temporal" dimension, we exploit strong signal correlations between successive observations in the sequence to reduce error rates. We evaluate SSG2 across three diverse datasets: UrbanMonitor, featuring orthoimage tiles from Darwin, Australia with five spectral bands and 0.2m spatial resolution; ISPRS Potsdam, which includes true orthophoto images with multiple spectral bands and a 5cm ground sampling
  arXiv  Detail & Related papers  (2023-10-12T19:08:03Z)
• Using a Conditional Generative Adversarial Network to Control the Statistical Characteristics of Generated Images for IACT Data Analysis [55.41644538483948]
  We divide images into several classes according to the value of some property of the image, and then specify the required class when generating new images. In the case of images from Imaging Atmospheric Cherenkov Telescopes (IACTs), an important property is the total brightness of all image pixels (image size) We used a cGAN technique to generate images similar to whose obtained in the TAIGA-IACT experiment.
  arXiv  Detail & Related papers  (2022-11-28T22:30:33Z)
• Multitask AET with Orthogonal Tangent Regularity for Dark Object Detection [84.52197307286681]
  We propose a novel multitask auto encoding transformation (MAET) model to enhance object detection in a dark environment. In a self-supervision manner, the MAET learns the intrinsic visual structure by encoding and decoding the realistic illumination-degrading transformation. We have achieved the state-of-the-art performance using synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-05-06T16:27:14Z)
• ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond [76.35955924137986]
  We propose a Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., ViTAE. ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context. We obtain the state-of-the-art classification performance, i.e., 88.5% Top-1 classification accuracy on ImageNet validation set and the best 91.2% Top-1 accuracy on ImageNet real validation set.
  arXiv  Detail & Related papers  (2022-02-21T10:40:05Z)
• Realistic galaxy image simulation via score-based generative models [0.0]
  We show that a score-based generative model can be used to produce realistic yet fake images that mimic observations of galaxies. Subjectively, the generated galaxies are highly realistic when compared with samples from the real dataset.
  arXiv  Detail & Related papers  (2021-11-02T16:27:08Z)
• There and Back Again: Self-supervised Multispectral Correspondence Estimation [13.56924750612194]
  We introduce a novel cycle-consistency metric that allows us to self-supervise. This, combined with our spectra-agnostic loss functions, allows us to train the same network across multiple spectra. We demonstrate our approach on the challenging task of dense RGB-FIR correspondence estimation.
  arXiv  Detail & Related papers  (2021-03-19T12:33:56Z)
• Cross-Spectral Periocular Recognition with Conditional Adversarial Networks [59.17685450892182]
  We propose Conditional Generative Adversarial Networks, trained to con-vert periocular images between visible and near-infrared spectra. We obtain a cross-spectral periocular performance of EER=1%, and GAR>99% @ FAR=1%, which is comparable to the state-of-the-art with the PolyU database.
  arXiv  Detail & Related papers  (2020-08-26T15:02:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.