Related papers: A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints

A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints

URL: http://arxiv.org/abs/2304.09941v2
Date: Thu, 31 Aug 2023 21:06:46 GMT
Title: A Robust and Interpretable Deep Learning Framework for Multi-modal Registration via Keypoints
Authors: Alan Q. Wang, Evan M. Yu, Adrian V. Dalca, Mert R. Sabuncu
Abstract summary: We present KeyMorph, a deep learning-based image registration framework. KeyMorph relies on automatically detecting corresponding keypoints. We show registration accuracy that surpasses current state-of-the-art methods.
Score: 10.913822141584795
License: http://creativecommons.org/publicdomain/zero/1.0/
Abstract: We present KeyMorph, a deep learning-based image registration framework that relies on automatically detecting corresponding keypoints. State-of-the-art deep learning methods for registration often are not robust to large misalignments, are not interpretable, and do not incorporate the symmetries of the problem. In addition, most models produce only a single prediction at test-time. Our core insight which addresses these shortcomings is that corresponding keypoints between images can be used to obtain the optimal transformation via a differentiable closed-form expression. We use this observation to drive the end-to-end learning of keypoints tailored for the registration task, and without knowledge of ground-truth keypoints. This framework not only leads to substantially more robust registration but also yields better interpretability, since the keypoints reveal which parts of the image are driving the final alignment. Moreover, KeyMorph can be designed to be equivariant under image translations and/or symmetric with respect to the input image ordering. Finally, we show how multiple deformation fields can be computed efficiently and in closed-form at test time corresponding to different transformation variants. We demonstrate the proposed framework in solving 3D affine and spline-based registration of multi-modal brain MRI scans. In particular, we show registration accuracy that surpasses current state-of-the-art methods, especially in the context of large displacements. Our code is available at https://github.com/alanqrwang/keymorph.

Related papers

With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning [47.96387857237473]
We devise a network which can perform attention over activations obtained while processing other training samples. Our memory models the distribution of past keys and values through the definition of prototype vectors. We demonstrate that our proposal can increase the performance of an encoder-decoder Transformer by 3.7 CIDEr points both when training in cross-entropy only and when fine-tuning with self-critical sequence training.
arXiv Detail & Related papers (2023-08-23T18:53:00Z)
Learning Feature Matching via Matchable Keypoint-Assisted Graph Neural Network [52.29330138835208]
Accurately matching local features between a pair of images is a challenging computer vision task. Previous studies typically use attention based graph neural networks (GNNs) with fully-connected graphs over keypoints within/across images. We propose MaKeGNN, a sparse attention-based GNN architecture which bypasses non-repeatable keypoints and leverages matchable ones to guide message passing.
arXiv Detail & Related papers (2023-07-04T02:50:44Z)
Breaking Modality Disparity: Harmonized Representation for Infrared and Visible Image Registration [66.33746403815283]
We propose a scene-adaptive infrared and visible image registration. We employ homography to simulate the deformation between different planes. We propose the first ground truth available misaligned infrared and visible image dataset.
arXiv Detail & Related papers (2023-04-12T06:49:56Z)
Anatomy-aware and acquisition-agnostic joint registration with SynthMorph [6.017634371712142]
Affine image registration is a cornerstone of medical image analysis. Deep-learning (DL) methods learn a function that maps an image pair to an output transform. Most affine methods are agnostic to the anatomy the user wishes to align, meaning the registration will be inaccurate if algorithms consider all structures in the image. We address these shortcomings with SynthMorph, a fast, symmetric, diffeomorphic, and easy-to-use DL tool for joint affine-deformable registration of any brain image.
arXiv Detail & Related papers (2023-01-26T18:59:33Z)
Non-iterative Coarse-to-fine Registration based on Single-pass Deep Cumulative Learning [11.795108660250843]
We propose a Non-Iterative Coarse-to-finE registration network (NICE-Net) for deformable image registration. NICE-Net can outperform state-of-the-art iterative deep registration methods while only requiring similar runtime to non-iterative methods.
arXiv Detail & Related papers (2022-06-25T08:34:59Z)
Self-Supervised Equivariant Learning for Oriented Keypoint Detection [35.94215211409985]
We introduce a self-supervised learning framework using rotation-equivariant CNNs to learn to detect robust oriented keypoints. We propose a dense orientation alignment loss by an image pair generated by synthetic transformations for training a histogram-based orientation map. Our method outperforms the previous methods on an image matching benchmark and a camera pose estimation benchmark.
arXiv Detail & Related papers (2022-04-19T02:26:07Z)
A Learning Framework for Diffeomorphic Image Registration based on Quasi-conformal Geometry [1.2891210250935146]
We propose the quasi-conformal registration network (QCRegNet), an unsupervised learning framework, to obtain diffeomorphic 2D image registrations. QCRegNet consists of the estimator network and the Beltrami solver network (BSNet) Results show that the registration accuracy is comparable to state-of-the-art methods and diffeomorphism is to a great extent guaranteed.
arXiv Detail & Related papers (2021-10-20T14:23:24Z)
Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration [67.69257782645789]
We propose piecewise transformation fields that learn 3D translation vectors to map any query point in posed space to its correspond position in rest-pose space. We show that fitting parametric models with poses by our network results in much better registration quality, especially for extreme poses.
arXiv Detail & Related papers (2021-04-16T15:16:09Z)
Test-Time Training for Deformable Multi-Scale Image Registration [15.523457398508263]
Deep learning-based registration approaches such as VoxelMorph have been emerging and achieve competitive performance. We construct a test-time training for deep deformable image registration to improve the generalization ability of conventional learning-based registration model.
arXiv Detail & Related papers (2021-03-25T03:22:59Z)
Shape My Face: Registering 3D Face Scans by Surface-to-Surface Translation [75.59415852802958]
Shape-My-Face (SMF) is a powerful encoder-decoder architecture based on an improved point cloud encoder, a novel visual attention mechanism, graph convolutional decoders with skip connections, and a specialized mouth model. Our model provides topologically-sound meshes with minimal supervision, offers faster training time, has orders of magnitude fewer trainable parameters, is more robust to noise, and can generalize to previously unseen datasets.
arXiv Detail & Related papers (2020-12-16T20:02:36Z)
Recurrent Multi-view Alignment Network for Unsupervised Surface Registration [79.72086524370819]
Learning non-rigid registration in an end-to-end manner is challenging due to the inherent high degrees of freedom and the lack of labeled training data. We propose to represent the non-rigid transformation with a point-wise combination of several rigid transformations. We also introduce a differentiable loss function that measures the 3D shape similarity on the projected multi-view 2D depth images.
arXiv Detail & Related papers (2020-11-24T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.