Aligning Motion-Blurred Images Using Contrastive Learning on Overcomplete Pixels

• URL: http://arxiv.org/abs/2410.07410v2
• Date: Fri, 1 Nov 2024 16:34:04 GMT
• Title: Aligning Motion-Blurred Images Using Contrastive Learning on Overcomplete Pixels
• Authors: Leonid Pogorelyuk, Stefan T. Radev,
• Abstract summary: We propose a new contrastive objective for learning overcomplete pixel-level features that are invariant to motion blur. We showcase that a simple U-Net trained with our objective can produce local features useful for aligning the frames of an unseen video captured with a moving camera under realistic and challenging conditions.
• Score: 1.8810643529425775
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a new contrastive objective for learning overcomplete pixel-level features that are invariant to motion blur. Other invariances (e.g., pose, illumination, or weather) can be learned by applying the corresponding transformations on unlabeled images during self-supervised training. We showcase that a simple U-Net trained with our objective can produce local features useful for aligning the frames of an unseen video captured with a moving camera under realistic and challenging conditions. Using a carefully designed toy example, we also show that the overcomplete pixels can encode the identity of objects in an image and the pixel coordinates relative to these objects.

Related papers

• Rethinking Image-to-Video Adaptation: An Object-centric Perspective [61.833533295978484]
  We propose a novel and efficient image-to-video adaptation strategy from the object-centric perspective. Inspired by human perception, we integrate a proxy task of object discovery into image-to-video transfer learning.
  arXiv  Detail & Related papers  (2024-07-09T13:58:10Z)
• Learning Fine-Grained Features for Pixel-wise Video Correspondences [13.456993858078514]
  We address the problem of learning features for establishing pixel-wise correspondences. Motivated by optical flows as well as the self-supervised feature learning, we propose to use not only labeled synthetic videos but also unlabeled real-world videos. Our experimental results on a series of correspondence-based tasks demonstrate that the proposed method outperforms state-of-the-art rivals in both accuracy and efficiency.
  arXiv  Detail & Related papers  (2023-08-06T07:27:17Z)
• Pixel-level Correspondence for Self-Supervised Learning from Video [56.24439897867531]
  Pixel-level Correspondence (PiCo) is a method for dense contrastive learning from video. We validate PiCo on standard benchmarks, outperforming self-supervised baselines on multiple dense prediction tasks.
  arXiv  Detail & Related papers  (2022-07-08T12:50:13Z)
• Real-Time Neural Character Rendering with Pose-Guided Multiplane Images [75.62730144924566]
  We propose pose-guided multiplane image (MPI) synthesis which can render an animatable character in real scenes with photorealistic quality. We use a portable camera rig to capture the multi-view images along with the driving signal for the moving subject.
  arXiv  Detail & Related papers  (2022-04-25T17:51:38Z)
• Self-Supervised Representation Learning from Flow Equivariance [97.13056332559526]
  We present a new self-supervised learning representation framework that can be directly deployed on a video stream of complex scenes. Our representations, learned from high-resolution raw video, can be readily used for downstream tasks on static images.
  arXiv  Detail & Related papers  (2021-01-16T23:44:09Z)
• Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation [67.88276573341734]
  We propose a new method for unseen object instance segmentation by learning RGB-D feature embeddings from synthetic data. A metric learning loss function is utilized to learn to produce pixel-wise feature embeddings. We further improve the segmentation accuracy with a new two-stage clustering algorithm.
  arXiv  Detail & Related papers  (2020-07-30T00:23:07Z)
• Self-Supervised Linear Motion Deblurring [112.75317069916579]
  Deep convolutional neural networks are state-of-the-art for image deblurring. We present a differentiable reblur model for self-supervised motion deblurring. Our experiments demonstrate that self-supervised single image deblurring is really feasible.
  arXiv  Detail & Related papers  (2020-02-10T20:15:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.