Uncertainty-Aware Cross-Modal Transfer Network for Sketch-Based 3D Shape Retrieval

• URL: http://arxiv.org/abs/2308.05948v1
• Date: Fri, 11 Aug 2023 05:46:52 GMT
• Title: Uncertainty-Aware Cross-Modal Transfer Network for Sketch-Based 3D Shape Retrieval
• Authors: Yiyang Cai, Jiaming Lu, Jiewen Wang, Shuang Liang
• Abstract summary: This paper presents an uncertainty-aware cross-modal transfer network (UACTN) that addresses this issue. We first introduce an end-to-end classification-based approach that simultaneously learns sketch features and uncertainty. Then, 3D shape features are mapped into the pre-learned sketch embedding space for feature alignment.
• Score: 8.765045867163646
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, sketch-based 3D shape retrieval has attracted growing attention. While many previous studies have focused on cross-modal matching between hand-drawn sketches and 3D shapes, the critical issue of how to handle low-quality and noisy samples in sketch data has been largely neglected. This paper presents an uncertainty-aware cross-modal transfer network (UACTN) that addresses this issue. UACTN decouples the representation learning of sketches and 3D shapes into two separate tasks: classification-based sketch uncertainty learning and 3D shape feature transfer. We first introduce an end-to-end classification-based approach that simultaneously learns sketch features and uncertainty, allowing uncertainty to prevent overfitting noisy sketches by assigning different levels of importance to clean and noisy sketches. Then, 3D shape features are mapped into the pre-learned sketch embedding space for feature alignment. Extensive experiments and ablation studies on two benchmarks demonstrate the superiority of our proposed method compared to state-of-the-art methods.

Related papers

• Multi-Style Facial Sketch Synthesis through Masked Generative Modeling [17.313050611750413]
  We propose a lightweight end-to-end synthesis model that efficiently converts images to corresponding multi-stylized sketches. In this study, we overcome the issue of data insufficiency by incorporating semi-supervised learning into the training process. Our method consistently outperforms previous algorithms across multiple benchmarks.
  arXiv  Detail & Related papers  (2024-08-22T13:45:04Z)
• Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation [55.73399465968594]
  This paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss.
  arXiv  Detail & Related papers  (2024-04-02T11:03:24Z)
• FILP-3D: Enhancing 3D Few-shot Class-incremental Learning with Pre-trained Vision-Language Models [62.663113296987085]
  Few-shot class-incremental learning aims to mitigate the catastrophic forgetting issue when a model is incrementally trained on limited data. We introduce two novel components: the Redundant Feature Eliminator (RFE) and the Spatial Noise Compensator (SNC) Considering the imbalance in existing 3D datasets, we also propose new evaluation metrics that offer a more nuanced assessment of a 3D FSCIL model.
  arXiv  Detail & Related papers  (2023-12-28T14:52:07Z)
• Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes [118.406721663244]
  We introduce a novel part-level modelling and alignment framework that facilitates abstraction modelling and cross-modal correspondence. Our approach seamlessly extends to sketch modelling by establishing correspondence between CLIPasso edgemaps and projected 3D part regions.
  arXiv  Detail & Related papers  (2023-12-07T05:04:33Z)
• Cross-modal and Cross-domain Knowledge Transfer for Label-free 3D Segmentation [23.110443633049382]
  We propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds. Our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on Semantic KITTI by using the knowledge of KITTI360 and GTA5.
  arXiv  Detail & Related papers  (2023-09-19T14:29:57Z)
• Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation [13.47191379827792]
  We investigate how large pre-trained models can be used to generate 3D shapes from sketches. We find that conditioning a 3D generative model on the features of synthetic renderings during training enables us to effectively generate 3D shapes from sketches at inference time. This suggests that the large pre-trained vision model features carry semantic signals that are resilient to domain shifts.
  arXiv  Detail & Related papers  (2023-07-08T00:45:01Z)
• Joint Deep Multi-Graph Matching and 3D Geometry Learning from Inhomogeneous 2D Image Collections [57.60094385551773]
  We propose a trainable framework for learning a deformable 3D geometry model from inhomogeneous image collections. We in addition obtain the underlying 3D geometry of the objects depicted in the 2D images.
  arXiv  Detail & Related papers  (2021-03-31T17:25:36Z)
• Deep Self-Supervised Representation Learning for Free-Hand Sketch [51.101565480583304]
  We tackle the problem of self-supervised representation learning for free-hand sketches. Key for the success of our self-supervised learning paradigm lies with our sketch-specific designs. We show that the proposed approach outperforms the state-of-the-art unsupervised representation learning methods.
  arXiv  Detail & Related papers  (2020-02-03T16:28:29Z)
• SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence [68.63311821718416]
  We study the problem of multi-view sketch correspondence, where we take as input multiple freehand sketches with different views of the same object. This problem is challenging since the visual features of corresponding points at different views can be very different. We take a deep learning approach and learn a novel local sketch descriptor from data.
  arXiv  Detail & Related papers  (2020-01-16T11:31:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.