Optimizing Multi-Modal Models for Image-Based Shape Retrieval: The Role of Pre-Alignment and Hard Contrastive Learning

• URL: http://arxiv.org/abs/2603.06982v1
• Date: Sat, 07 Mar 2026 01:54:35 GMT
• Title: Optimizing Multi-Modal Models for Image-Based Shape Retrieval: The Role of Pre-Alignment and Hard Contrastive Learning
• Abstract summary: Image-based shape retrieval (IBSR) aims to retrieve 3D models from a database given a query image.<n>We address IBSR through large-scale multi-modal pretraining and show that explicit view-based supervision is not required.
• Score: 8.222080530754223
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Image-based shape retrieval (IBSR) aims to retrieve 3D models from a database given a query image, hence addressing a classical task in computer vision, computer graphics, and robotics. Recent approaches typically rely on bridging the domain gap between 2D images and 3D shapes based on the use of multi-view renderings as well as task-specific metric learning to embed shapes and images into a common latent space. In contrast, we address IBSR through large-scale multi-modal pretraining and show that explicit view-based supervision is not required. Inspired by pre-aligned image--point-cloud encoders from ULIP and OpenShape that have been used for tasks such as 3D shape classification, we propose the use of pre-aligned image and shape encoders for zero-shot and standard IBSR by embedding images and point clouds into a shared representation space and performing retrieval via similarity search over compact single-embedding shape descriptors. This formulation allows skipping view synthesis and naturally enables zero-shot and cross-domain retrieval without retraining on the target database. We evaluate pre-aligned encoders in both zero-shot and supervised IBSR settings and additionally introduce a multi-modal hard contrastive loss (HCL) to further increase retrieval performance. Our evaluation demonstrates state-of-the-art performance, outperforming related methods on $Acc_{Top1}$ and $Acc_{Top10}$ for shape retrieval across multiple datasets, with best results observed for OpenShape combined with Point-BERT. Furthermore, training on our proposed multi-modal HCL yields dataset-dependent gains in standard instance retrieval tasks on shape-centric data, underscoring the value of pretraining and hard contrastive learning for 3D shape retrieval. The code will be made available via the project website.

Related papers

• Evaluation of Visual Place Recognition Methods for Image Pair Retrieval in 3D Vision and Robotics [6.59608906817608]
  We study Visual Place Recognition (VPR) as an image pair retrieval front-end for registration pipelines.<n>We comparatively evaluate state-of-the-art VPR families on three challenging datasets.
  arXiv  Detail & Related papers  (2026-03-14T12:16:31Z)
• OPFormer: Object Pose Estimation leveraging foundation model with geometric encoding [2.1987601456703474]
  We introduce a unified, end-to-end framework that seamlessly integrates object detection and pose estimation.<n>Our system first employs the CNOS detector to localize target objects.<n>For each detection, our novel pose estimation module, OPFormer, infers the precise 6D pose.
  arXiv  Detail & Related papers  (2025-11-16T14:19:52Z)
• Beyond 'Templates': Category-Agnostic Object Pose, Size, and Shape Estimation from a Single View [69.6117755984012]
  Estimating an object's 6D pose, size, and shape from visual input is a fundamental problem in computer vision.<n>We propose a unified category-agnostic framework that simultaneously predicts 6D pose, size, and dense shape from a single RGB-D image.
  arXiv  Detail & Related papers  (2025-10-13T17:49:15Z)
• Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval [76.86914849263168]
  Open-set 3D object retrieval is an emerging task aiming to retrieve 3D objects of unseen categories beyond the training set.<n>Existing methods typically utilize all modalities (i.e., voxels, point clouds, multi-view images) and train specific backbones before fusion.<n>We present a framework named Describe, Adapt and Combine (DAC) by taking only multi-view images for open-set 3DOR.
  arXiv  Detail & Related papers  (2025-07-29T04:11:05Z)
• Revisiting CLIP: Efficient Alignment of 3D MRI and Tabular Data using Domain-Specific Foundation Models [2.0749231618270803]
  We revisit CLIP-style alignment by training a domain-specific 3D foundation model as an image encoder.<n>Our approach is enabled by a simple embedding accumulation strategy required for training in 3D.
  arXiv  Detail & Related papers  (2025-01-23T19:34:48Z)
• Anyview: Generalizable Indoor 3D Object Detection with Variable Frames [60.48134767838629]
  We present a novel 3D detection framework named AnyView for our practical applications.<n>Our method achieves both great generalizability and high detection accuracy with a simple and clean architecture.
  arXiv  Detail & Related papers  (2023-10-09T02:15:45Z)
• Geometric-aware Pretraining for Vision-centric 3D Object Detection [77.7979088689944]
  We propose a novel geometric-aware pretraining framework called GAPretrain. GAPretrain serves as a plug-and-play solution that can be flexibly applied to multiple state-of-the-art detectors. We achieve 46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with a gain of 2.7 and 2.1 points, respectively.
  arXiv  Detail & Related papers  (2023-04-06T14:33:05Z)
• Semantic keypoint-based pose estimation from single RGB frames [64.80395521735463]
  We present an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. We show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios.
  arXiv  Detail & Related papers  (2022-04-12T15:03:51Z)
• Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training [3.8073142980733]
  We propose a novel framework for monocular 3D objects detection using only RGB images, called KM3D-Net. We design a fully convolutional model to predict object keypoints, dimension, and orientation, and then combine these estimations with perspective geometry constraints to compute position attribute.
  arXiv  Detail & Related papers  (2020-09-02T00:51:51Z)
• Two-shot Spatially-varying BRDF and Shape Estimation [89.29020624201708]
  We propose a novel deep learning architecture with a stage-wise estimation of shape and SVBRDF. We create a large-scale synthetic training dataset with domain-randomized geometry and realistic materials. Experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images.
  arXiv  Detail & Related papers  (2020-04-01T12:56:13Z)
• Object Detection on Single Monocular Images through Canonical Correlation Analysis [3.4722706398428493]
  We retrieve 3-D object information from single monocular images without using extra 3-D data like points cloud or depth images. We propose a two-dimensional CCA framework to fuse monocular images and corresponding predicted depth images for basic computer vision tasks.
  arXiv  Detail & Related papers  (2020-02-13T05:03:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.