Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning

• URL: http://arxiv.org/abs/2412.19101v1
• Date: Thu, 26 Dec 2024 07:43:01 GMT
• Title: Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning
• Authors: Ran Ma, Yixiong Zou, Yuhua Li, Ruixuan Li,
• Abstract summary: Cross-Domain Few-Shot Learning requires the model to transfer knowledge from the data-abundant source domain to data-scarce target domains. Masked Autoencoder (MAE) excels in effectively using unlabeled data and learning image's global structures. We propose Domain-Agnostic Masked Image Modeling (DAMIM) for the Cross-Domain Few-Shot Learning task.
• Score: 10.384302950402969
• License:
• Abstract: Cross-Domain Few-Shot Learning (CDFSL) requires the model to transfer knowledge from the data-abundant source domain to data-scarce target domains for fast adaptation, where the large domain gap makes CDFSL a challenging problem. Masked Autoencoder (MAE) excels in effectively using unlabeled data and learning image's global structures, enhancing model generalization and robustness. However, in the CDFSL task with significant domain shifts, we find MAE even shows lower performance than the baseline supervised models. In this paper, we first delve into this phenomenon for an interpretation. We find that MAE tends to focus on low-level domain information during reconstructing pixels while changing the reconstruction target to token features could mitigate this problem. However, not all features are beneficial, as we then find reconstructing high-level features can hardly improve the model's transferability, indicating a trade-off between filtering domain information and preserving the image's global structure. In all, the reconstruction target matters for the CDFSL task. Based on the above findings and interpretations, we further propose Domain-Agnostic Masked Image Modeling (DAMIM) for the CDFSL task. DAMIM includes an Aggregated Feature Reconstruction module to automatically aggregate features for reconstruction, with balanced learning of domain-agnostic information and images' global structure, and a Lightweight Decoder module to further benefit the encoder's generalizability. Experiments on four CDFSL datasets demonstrate that our method achieves state-of-the-art performance.

Related papers

• A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gap [50.079224604394]
  We present a novel model-agnostic framework called textbfContext-textbfEnhanced textbfFeature textbfAment (CEFA) CEFA consists of a feature alignment module and a context enhancement module. Our method can serve as a plug-and-play module to improve the detection performance of HOI models on rare categories.
  arXiv  Detail & Related papers  (2024-07-31T08:42:48Z)
• Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment [20.902935570581207]
  We introduce a Multimodal Alignment and Reconstruction Network (MARNet) to enhance the model's resistance to visual noise. MARNet includes a cross-modal diffusion reconstruction module for smoothly and stably blending information across different domains. Experiments conducted on two benchmark datasets, Vireo-Food172 and Ingredient-101, demonstrate that MARNet effectively improves the quality of image information extracted by the model.
  arXiv  Detail & Related papers  (2024-07-26T16:30:18Z)
• IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions [26.09373405194564]
  We present an efficient image processing transformer architecture with hierarchical attentions, called IPTV2. We adopt a focal context self-attention (FCSA) and a global grid self-attention (GGSA) to obtain adequate token interactions in local and global receptive fields. Our proposed IPT-V2 achieves state-of-the-art results on various image processing tasks, covering denoising, deblurring, deraining and obtains much better trade-off for performance and computational complexity than previous methods.
  arXiv  Detail & Related papers  (2024-03-31T10:01:20Z)
• In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
  We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model. We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
  arXiv  Detail & Related papers  (2023-09-25T08:42:06Z)
• Enhancing Low-light Light Field Images with A Deep Compensation Unfolding Network [52.77569396659629]
  This paper presents the deep compensation network unfolding (DCUNet) for restoring light field (LF) images captured under low-light conditions. The framework uses the intermediate enhanced result to estimate the illumination map, which is then employed in the unfolding process to produce a new enhanced result. To properly leverage the unique characteristics of LF images, this paper proposes a pseudo-explicit feature interaction module.
  arXiv  Detail & Related papers  (2023-08-10T07:53:06Z)
• Autoencoders with Intrinsic Dimension Constraints for Learning Low Dimensional Image Representations [27.40298734517967]
  We propose a novel deep representation learning approach with autoencoder, which incorporates regularization of the global and local ID constraints into the reconstruction of data representations. This approach not only preserves the global manifold structure of the whole dataset, but also maintains the local manifold structure of the feature maps of each point.
  arXiv  Detail & Related papers  (2023-04-16T03:43:08Z)
• CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training [112.96224800952724]
  We propose cascaded modulation GAN (CM-GAN) to generate plausible image structures when dealing with large holes in complex images. In each decoder block, global modulation is first applied to perform coarse semantic-aware synthesis structure, then spatial modulation is applied on the output of global modulation to further adjust the feature map in a spatially adaptive fashion. In addition, we design an object-aware training scheme to prevent the network from hallucinating new objects inside holes, fulfilling the needs of object removal tasks in real-world scenarios.
  arXiv  Detail & Related papers  (2022-03-22T16:13:27Z)
• Specificity-Preserving Federated Learning for MR Image Reconstruction [94.58912814426122]
  Federated learning can be used to improve data privacy and efficiency in magnetic resonance (MR) image reconstruction. Recent FL techniques tend to solve this by enhancing the generalization of the global model. We propose a specificity-preserving FL algorithm for MR image reconstruction (FedMRI)
  arXiv  Detail & Related papers  (2021-12-09T22:13:35Z)
• Boosting Few-shot Semantic Segmentation with Transformers [81.43459055197435]
  TRansformer-based Few-shot Semantic segmentation method (TRFS) Our model consists of two modules: Global Enhancement Module (GEM) and Local Enhancement Module (LEM)
  arXiv  Detail & Related papers  (2021-08-04T20:09:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.