AstroMAE: Redshift Prediction Using a Masked Autoencoder with a Novel Fine-Tuning Architecture

• URL: http://arxiv.org/abs/2409.01825v1
• Date: Tue, 3 Sep 2024 12:12:37 GMT
• Title: AstroMAE: Redshift Prediction Using a Masked Autoencoder with a Novel Fine-Tuning Architecture
• Authors: Amirreza Dolatpour Fathkouhi, Geoffrey Charles Fox,
• Abstract summary: We introduce AstroMAE, an innovative approach that pretrains a vision transformer encoder using a masked autoencoder method. This technique enables the encoder to capture the global patterns within the data without relying on labels. We evaluate our model against various vision transformer architectures and CNN-based models.
• Score: 0.6906005491572401
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Redshift prediction is a fundamental task in astronomy, essential for understanding the expansion of the universe and determining the distances of astronomical objects. Accurate redshift prediction plays a crucial role in advancing our knowledge of the cosmos. Machine learning (ML) methods, renowned for their precision and speed, offer promising solutions for this complex task. However, traditional ML algorithms heavily depend on labeled data and task-specific feature extraction. To overcome these limitations, we introduce AstroMAE, an innovative approach that pretrains a vision transformer encoder using a masked autoencoder method on Sloan Digital Sky Survey (SDSS) images. This technique enables the encoder to capture the global patterns within the data without relying on labels. To the best of our knowledge, AstroMAE represents the first application of a masked autoencoder to astronomical data. By ignoring labels during the pretraining phase, the encoder gathers a general understanding of the data. The pretrained encoder is subsequently fine-tuned within a specialized architecture tailored for redshift prediction. We evaluate our model against various vision transformer architectures and CNN-based models, demonstrating the superior performance of AstroMAEs pretrained model and fine-tuning architecture.

Related papers

• Disentangling stellar atmospheric parameters in astronomical spectra using Generative Adversarial Neural Networks [0.7758482228293941]
  A method based on Generative Adversaria! Networks (GANs) is developed for disentangling the physical (effective temperature and gravity) and chemical (metallicity, overabundance of a-elements with respect to iron) atmospheric properties in astronomical spectra.
  arXiv  Detail & Related papers  (2025-01-20T21:53:34Z)
• Understanding Encoder-Decoder Structures in Machine Learning Using Information Measures [10.066310107046084]
  We present new results to model and understand the role of encoder-decoder design in machine learning (ML) We use two main information concepts, information sufficiency (IS) and mutual information loss (MIL), to represent predictive structures in machine learning.
  arXiv  Detail & Related papers  (2024-05-30T19:58:01Z)
• Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning [18.10704604275133]
  Masked Autoencoders (MAE) have demonstrated promising performance in self-supervised learning for 2D and 3D computer vision. We propose Point Regress AutoEncoder (Point-RAE), a new scheme for regressive autoencoders for point cloud self-supervised learning. Our approach is efficient during pre-training and generalizes well on various downstream tasks.
  arXiv  Detail & Related papers  (2023-09-25T17:23:33Z)
• ARFA: An Asymmetric Receptive Field Autoencoder Model for Spatiotemporal Prediction [55.30913411696375]
  We propose an Asymmetric Receptive Field Autoencoder (ARFA) model, which introduces corresponding sizes of receptive field modules. In the encoder, we present large kernel module for globaltemporal feature extraction. In the decoder, we develop a small kernel module for localtemporal reconstruction. We construct the RainBench, a large-scale radar echo dataset for precipitation prediction, to address the scarcity of meteorological data in the domain.
  arXiv  Detail & Related papers  (2023-09-01T07:55:53Z)
• Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving [74.28510044056706]
  Existing methods usually adopt the decoupled encoder-decoder paradigm. In this work, we aim to alleviate the problem by two principles. We first predict a coarse-grained future position and action based on the encoder features. Then, conditioned on the position and action, the future scene is imagined to check the ramification if we drive accordingly.
  arXiv  Detail & Related papers  (2023-05-10T15:22:02Z)
• MAPLE: Masked Pseudo-Labeling autoEncoder for Semi-supervised Point Cloud Action Recognition [160.49403075559158]
  We propose a Masked Pseudo-Labeling autoEncoder (textbfMAPLE) framework for point cloud action recognition. In particular, we design a novel and efficient textbfDecoupled textbfspatial-textbftemporal TranstextbfFormer (textbfDestFormer) as the backbone of MAPLE. MAPLE achieves superior results on three public benchmarks and outperforms the state-of-the-art method by 8.08% accuracy on the MSR-Action3
  arXiv  Detail & Related papers  (2022-09-01T12:32:40Z)
• Context Autoencoder for Self-Supervised Representation Learning [64.63908944426224]
  We pretrain an encoder by making predictions in the encoded representation space. The network is an encoder-regressor-decoder architecture. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks.
  arXiv  Detail & Related papers  (2022-02-07T09:33:45Z)
• Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems [38.672160430296536]
  Transformer models have achieved state-of-the-art results in a wide range of NLP tasks including summarization. Previous work has focused on one important bottleneck, the quadratic self-attention mechanism in the encoder. This work focuses on the transformer's encoder-decoder attention mechanism.
  arXiv  Detail & Related papers  (2021-09-08T19:32:42Z)
• Learning Spatio-Temporal Transformer for Visual Tracking [108.11680070733598]
  We present a new tracking architecture with an encoder-decoder transformer as the key component. The whole method is end-to-end, does not need any postprocessing steps such as cosine window and bounding box smoothing. The proposed tracker achieves state-of-the-art performance on five challenging short-term and long-term benchmarks, while running real-time speed, being 6x faster than Siam R-CNN.
  arXiv  Detail & Related papers  (2021-03-31T15:19:19Z)
• Cross-Thought for Sentence Encoder Pre-training [89.32270059777025]
  Cross-Thought is a novel approach to pre-training sequence encoder. We train a Transformer-based sequence encoder over a large set of short sequences. Experiments on question answering and textual entailment tasks demonstrate that our pre-trained encoder can outperform state-of-the-art encoders.
  arXiv  Detail & Related papers  (2020-10-07T21:02:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.