Eliminating Warping Shakes for Unsupervised Online Video Stitching

• URL: http://arxiv.org/abs/2403.06378v2
• Date: Wed, 10 Jul 2024 09:05:54 GMT
• Title: Eliminating Warping Shakes for Unsupervised Online Video Stitching
• Authors: Lang Nie, Chunyu Lin, Kang Liao, Yun Zhang, Shuaicheng Liu, Rui Ai, Yao Zhao,
• Abstract summary: We retarget video stitching to an emerging issue, named warping shake, when extending image stitching to video stitching. We propose StabStitch to simultaneously realize video stitching and video stabilization in a unified unsupervised learning framework. Compared with existing stitching solutions, StabStitch exhibits significant superiority in scene robustness and inference speed in addition to stitching and stabilization performance.
• Score: 77.96880800111498
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we retarget video stitching to an emerging issue, named warping shake, when extending image stitching to video stitching. It unveils the temporal instability of warped content in non-overlapping regions, despite image stitching having endeavored to preserve the natural structures. Therefore, in most cases, even if the input videos to be stitched are stable, the stitched video will inevitably cause undesired warping shakes and affect the visual experience. To eliminate the shakes, we propose StabStitch to simultaneously realize video stitching and video stabilization in a unified unsupervised learning framework. Starting from the camera paths in video stabilization, we first derive the expression of stitching trajectories in video stitching by elaborately integrating spatial and temporal warps. Then a warp smoothing model is presented to optimize them with a comprehensive consideration regarding content alignment, trajectory smoothness, spatial consistency, and online collaboration. To establish an evaluation benchmark and train the learning framework, we build a video stitching dataset with a rich diversity in camera motions and scenes. Compared with existing stitching solutions, StabStitch exhibits significant superiority in scene robustness and inference speed in addition to stitching and stabilization performance, contributing to a robust and real-time online video stitching system. The code and dataset are available at https://github.com/nie-lang/StabStitch.

Related papers

• Video Demoireing using Focused-Defocused Dual-Camera System [21.59133575445115]
  Existing demoireing methods rely on single-camera image/video processing.<n>We propose a dual-camera framework that captures synchronized videos of the same scene.<n>We use the defocused video to help distinguish moire patterns from real texture, so as to guide the demoireing of the focused video.
  arXiv  Detail & Related papers  (2025-08-05T13:49:49Z)
• GaVS: 3D-Grounded Video Stabilization via Temporally-Consistent Local Reconstruction and Rendering [54.489285024494855]
  Video stabilization is pivotal for video processing, as it removes unwanted shakiness while preserving the original user motion intent.<n>Existing approaches, depending on the domain they operate, suffer from several issues that degrade the user experience.<n>We introduce textbfGaVS, a novel 3D-grounded approach that reformulates video stabilization as a temporally-consistent local reconstruction and rendering' paradigm.
  arXiv  Detail & Related papers  (2025-06-30T15:24:27Z)
• StabStitch++: Unsupervised Online Video Stitching with Spatiotemporal Bidirectional Warps [81.8786100662034]
  We retarget video stitching to an emerging issue, named warping shake, which unveils the temporal content shakes induced by sequentially unsmooth warps when extending image stitching to video stitching.<n>To address this issue, we propose StabStitch++, a novel video stitching framework to realize spatial stitching and temporal stabilization with unsupervised learning simultaneously.
  arXiv  Detail & Related papers  (2025-05-08T07:12:23Z)
• SketchVideo: Sketch-based Video Generation and Editing [51.99066098393491]
  We aim to achieve sketch-based spatial and motion control for video generation and support fine-grained editing of real or synthetic videos. Based on the DiT video generation model, we propose a memory-efficient control structure with sketch control blocks that predict residual features of skipped DiT blocks. For sketch-based video editing, we design an additional video insertion module that maintains consistency between the newly edited content and the original video's spatial feature and dynamic motion.
  arXiv  Detail & Related papers  (2025-03-30T02:44:09Z)
• VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion Control [66.66226299852559]
  VideoAnydoor is a zero-shot video object insertion framework with high-fidelity detail preservation and precise motion control. To preserve the detailed appearance and meanwhile support fine-grained motion control, we design a pixel warper.
  arXiv  Detail & Related papers  (2025-01-02T18:59:54Z)
• Task Agnostic Restoration of Natural Video Dynamics [10.078712109708592]
  In many video restoration/translation tasks, image processing operations are na"ively extended to the video domain by processing each frame independently. We propose a general framework for this task that learns to infer and utilize consistent motion dynamics from inconsistent videos to mitigate the temporal flicker. The proposed framework produces SOTA results on two benchmark datasets, DAVIS and videvo.net, processed by numerous image processing applications.
  arXiv  Detail & Related papers  (2022-06-08T09:00:31Z)
• Video Demoireing with Relation-Based Temporal Consistency [68.20281109859998]
  Moire patterns, appearing as color distortions, severely degrade image and video qualities when filming a screen with digital cameras. We study how to remove such undesirable moire patterns in videos, namely video demoireing.
  arXiv  Detail & Related papers  (2022-04-06T17:45:38Z)
• Self-Supervised Real-time Video Stabilization [100.00816752529045]
  We propose a novel method of real-time video stabilization. It transforms a shaky video to a stabilized video as if it were stabilized via gimbals in real-time.
  arXiv  Detail & Related papers  (2021-11-10T22:49:56Z)
• Attention-guided Temporal Coherent Video Object Matting [78.82835351423383]
  We propose a novel deep learning-based object matting method that can achieve temporally coherent matting results. Its key component is an attention-based temporal aggregation module that maximizes image matting networks' strength. We show how to effectively solve the trimap generation problem by fine-tuning a state-of-the-art video object segmentation network.
  arXiv  Detail & Related papers  (2021-05-24T17:34:57Z)
• Restoration of Video Frames from a Single Blurred Image with Motion Understanding [69.90724075337194]
  We propose a novel framework to generate clean video frames from a single motion-red image. We formulate video restoration from a single blurred image as an inverse problem by setting clean image sequence and their respective motion as latent factors. Our framework is based on anblur-decoder structure with spatial transformer network modules.
  arXiv  Detail & Related papers  (2021-04-19T08:32:57Z)
• Neural Re-rendering for Full-frame Video Stabilization [144.9918806873405]
  We present an algorithm for full-frame video stabilization by first estimating dense warp fields. Full-frame stabilized frames can then be synthesized by fusing warped contents from neighboring frames.
  arXiv  Detail & Related papers  (2021-02-11T18:59:45Z)
• A Sparse Sampling-based framework for Semantic Fast-Forward of First-Person Videos [2.362412515574206]
  Most uploaded videos are doomed to be forgotten and unwatched stashed away in some computer folder or website. We present a new adaptive frame selection formulated as a weighted minimum reconstruction problem. Our method is able to retain as much relevant information and smoothness as the state-of-the-art techniques, but in less processing time.
  arXiv  Detail & Related papers  (2020-09-21T18:36:17Z)
• Adaptively Meshed Video Stabilization [32.68960056325736]
  This paper proposes an adaptively meshed method to stabilize a shaky video based on all of its feature trajectories and an adaptive blocking strategy. We estimate the mesh-based transformations of each frame by solving a two-stage optimization problem.
  arXiv  Detail & Related papers  (2020-06-14T06:51:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.