GLIMPSE: Generalized Local Imaging with MLPs

• URL: http://arxiv.org/abs/2401.00816v2
• Date: Thu, 20 Jun 2024 14:36:36 GMT
• Title: GLIMPSE: Generalized Local Imaging with MLPs
• Authors: AmirEhsan Khorashadizadeh, Valentin Debarnot, Tianlin Liu, Ivan Dokmanić,
• Abstract summary: Deep learning is the current de facto state of the art in tomographic imaging. A common approach is to feed the result of a simple inversion to a convolutional neural network (CNN) which computes the reconstruction. We introduce GLIMPSE, a local processing neural network for computed tomography which reconstructs a pixel value by feeding only the measurements associated with the neighborhood of the pixel.
• Score: 10.657105348034753
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deep learning is the current de facto state of the art in tomographic imaging. A common approach is to feed the result of a simple inversion, for example the backprojection, to a convolutional neural network (CNN) which then computes the reconstruction. Despite strong results on 'in-distribution' test data similar to the training data, backprojection from sparse-view data delocalizes singularities, so these approaches require a large receptive field to perform well. As a consequence, they overfit to certain global structures which leads to poor generalization on out-of-distribution (OOD) samples. Moreover, their memory complexity and training time scale unfavorably with image resolution, making them impractical for application at realistic clinical resolutions, especially in 3D: a standard U-Net requires a substantial 140GB of memory and 2600 seconds per epoch on a research-grade GPU when training on 1024x1024 images. In this paper, we introduce GLIMPSE, a local processing neural network for computed tomography which reconstructs a pixel value by feeding only the measurements associated with the neighborhood of the pixel to a simple MLP. While achieving comparable or better performance with successful CNNs like the U-Net on in-distribution test data, GLIMPSE significantly outperforms them on OOD samples while maintaining a memory footprint almost independent of image resolution; 5GB memory suffices to train on 1024x1024 images. Further, we built GLIMPSE to be fully differentiable, which enables feats such as recovery of accurate projection angles if they are out of calibration.

Related papers

• Exploring Multi-view Pixel Contrast for General and Robust Image Forgery Localization [4.8454936010479335]
  We propose a Multi-view Pixel-wise Contrastive algorithm (MPC) for image forgery localization. Specifically, we first pre-train the backbone network with the supervised contrastive loss. Then the localization head is fine-tuned using the cross-entropy loss, resulting in a better pixel localizer.
  arXiv  Detail & Related papers  (2024-06-19T13:51:52Z)
• Parameter-Inverted Image Pyramid Networks [49.35689698870247]
  We propose a novel network architecture known as the Inverted Image Pyramid Networks (PIIP) Our core idea is to use models with different parameter sizes to process different resolution levels of the image pyramid. PIIP achieves superior performance in tasks such as object detection, segmentation, and image classification.
  arXiv  Detail & Related papers  (2024-06-06T17:59:10Z)
• Efficient Visual State Space Model for Image Deblurring [83.57239834238035]
  Convolutional neural networks (CNNs) and Vision Transformers (ViTs) have achieved excellent performance in image restoration. We propose a simple yet effective visual state space model (EVSSM) for image deblurring.
  arXiv  Detail & Related papers  (2024-05-23T09:13:36Z)
• Pixel-Inconsistency Modeling for Image Manipulation Localization [63.54342601757723]
  Digital image forensics plays a crucial role in image authentication and manipulation localization. This paper presents a generalized and robust manipulation localization model through the analysis of pixel inconsistency artifacts. Experiments show that our method successfully extracts inherent pixel-inconsistency forgery fingerprints.
  arXiv  Detail & Related papers  (2023-09-30T02:54:51Z)
• Memory Efficient Patch-based Training for INR-based GANs [13.19626131847784]
  Training existing approaches require a heavy computational cost proportional to the image resolution. We propose a multi-stage patch-based training, a novel and scalable approach that can train INR-based GANs with a flexible computational cost. Specifically, our method allows to generate and discriminate by patch to learn the local details of the image and learn global structural information.
  arXiv  Detail & Related papers  (2022-07-04T13:28:53Z)
• Class Balanced PixelNet for Neurological Image Segmentation [20.56747443955369]
  We propose an automatic brain tumor segmentation approach (e.g., PixelNet) using a pixel-level convolutional neural network (CNN) The proposed model has achieved promising results in brain tumor and ischemic stroke segmentation datasets.
  arXiv  Detail & Related papers  (2022-04-23T10:57:54Z)
• ACORN: Adaptive Coordinate Networks for Neural Scene Representation [40.04760307540698]
  Current neural representations fail to accurately represent images at resolutions greater than a megapixel or 3D scenes with more than a few hundred thousand polygons. We introduce a new hybrid implicit-explicit network architecture and training strategy that adaptively allocates resources during training and inference. We demonstrate the first experiments that fit gigapixel images to nearly 40 dB peak signal-to-noise ratio.
  arXiv  Detail & Related papers  (2021-05-06T16:21:38Z)
• PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layers [111.55817466296402]
  We introduce Perspective Crop Layers (PCLs) - a form of perspective crop of the region of interest based on the camera geometry. PCLs deterministically remove the location-dependent perspective effects while leaving end-to-end training and the number of parameters of the underlying neural network. PCL offers an easy way to improve the accuracy of existing 3D reconstruction networks by making them geometry aware.
  arXiv  Detail & Related papers  (2020-11-27T08:48:43Z)
• Locally Masked Convolution for Autoregressive Models [107.4635841204146]
  LMConv is a simple modification to the standard 2D convolution that allows arbitrary masks to be applied to the weights at each location in the image. We learn an ensemble of distribution estimators that share parameters but differ in generation order, achieving improved performance on whole-image density estimation.
  arXiv  Detail & Related papers  (2020-06-22T17:59:07Z)
• BP-DIP: A Backprojection based Deep Image Prior [49.375539602228415]
  We propose two image restoration approaches: (i) Deep Image Prior (DIP), which trains a convolutional neural network (CNN) from scratch in test time using the degraded image; and (ii) a backprojection (BP) fidelity term, which is an alternative to the standard least squares loss that is usually used in previous DIP works. We demonstrate the performance of the proposed method, termed BP-DIP, on the deblurring task and show its advantages over the plain DIP, with both higher PSNR values and better inference run-time.
  arXiv  Detail & Related papers  (2020-03-11T17:09:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.