EF-VI: Enhancing End-Frame Injection for Video Inbetweening

• URL: http://arxiv.org/abs/2505.21205v2
• Date: Sun, 10 Aug 2025 16:23:45 GMT
• Title: EF-VI: Enhancing End-Frame Injection for Video Inbetweening
• Abstract summary: Video inbetweening aims to synthesize intermediate video sequences conditioned on the given start and end frames.<n>We propose a novel video inbetweening framework specific to recent and more powerful transformer-based I2V-DMs, termed EF-VI.<n>It efficiently strengthens the end-frame constraint by utilizing an enhanced injection.
• Score: 52.6883373124261
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video inbetweening aims to synthesize intermediate video sequences conditioned on the given start and end frames. Current state-of-the-art methods primarily extend large-scale pre-trained Image-to-Video Diffusion Models (I2V-DMs) by incorporating the end-frame condition via direct fine-tuning or temporally bidirectional sampling. However, the former results in a weak end-frame constraint, while the latter inevitably disrupts the input representation of video frames, leading to suboptimal performance. To improve the end-frame constraint while avoiding disruption of the input representation, we propose a novel video inbetweening framework specific to recent and more powerful transformer-based I2V-DMs, termed EF-VI. It efficiently strengthens the end-frame constraint by utilizing an enhanced injection. This is based on our proposed well-designed lightweight module, termed EF-Net, which encodes only the end frame and expands it into temporally adaptive frame-wise features injected into the I2V-DM. Extensive experiments demonstrate the superiority of our EF-VI compared with other baselines.

Related papers

• FC-VFI: Faithful and Consistent Video Frame Interpolation for High-FPS Slow Motion Video Generation [14.850919655503871]
  We propose FC-VFI for faithful and consistent video frame preservation, supporting (4times)x and (8times)resolution.<n>We introduce a temporal modeling strategy on the latent sequences to inherit fidelity cues from start and end frames and leverage semantic matching lines for structure-aware motion guidance.
  arXiv  Detail & Related papers  (2026-03-05T07:41:34Z)
• Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers [95.68243351895107]
  We propose a holistic, video-centric paradigm named textbfLocal textbfDiffusion textbfForcing for textbfVideo textbfFrame textbfInterpolation (LDF-VFI)<n>Our framework is built upon an auto-regressive diffusion transformer that models the entire video sequence to ensure long-range temporal coherence.<n>LDF-VFI achieves state-of-the-art performance on challenging long-sequence benchmarks, demonstrating superior per
  arXiv  Detail & Related papers  (2026-01-21T12:58:52Z)
• EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation [95.60655992032316]
  We introduce EDEN, an Enhanced Diffusion for high-quality large-motion vidEo frame iNterpolation.<n>Our approach first utilizes a transformer-based tokenizer to produce refined latent representations of the intermediate frames for diffusion models.<n>We then enhance the diffusion transformer with temporal attention across the process and incorporate a start-end frame difference embedding to guide the generation of dynamic motion.
  arXiv  Detail & Related papers  (2025-03-20T03:54:52Z)
• Generative Inbetweening through Frame-wise Conditions-Driven Video Generation [63.43583844248389]
  generative inbetweening aims to generate intermediate frame sequences by utilizing two key frames as input.<n>We propose a Frame-wise Conditions-driven Video Generation (FCVG) method that significantly enhances the temporal stability of interpolated video frames.<n>Our FCVG demonstrates the capability to generate temporally stable videos using both linear and non-linear curves.
  arXiv  Detail & Related papers  (2024-12-16T13:19:41Z)
• ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler [53.98558445900626]
  Current image-to-video diffusion models, while powerful in generating videos from a single frame, need adaptation for two-frame conditioned generation.<n>We introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning.<n>Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames.
  arXiv  Detail & Related papers  (2024-10-08T03:01:54Z)
• A Multi-In-Single-Out Network for Video Frame Interpolation without Optical Flow [14.877766449009119]
  deep learning-based video frame (VFI) methods have predominantly focused on estimating motion between two input frames. We propose a multi-in-single-out (MISO) based VFI method that does not rely on motion vector estimation. We introduce a novel motion perceptual loss that enables MISO-VFI to better capture the vectors-temporal within the video frames.
  arXiv  Detail & Related papers  (2023-11-20T08:29:55Z)
• Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognition [9.803556181225193]
  Conformer as a backbone network for end-to-end automatic speech recognition achieved state-of-the-art performance. However, the Conformer-based model encounters an issue with the self-attention mechanism. We introduce key frame-based self-attention (KFSA) mechanism, a novel method to reduce the computation of the self-attention mechanism using key frames.
  arXiv  Detail & Related papers  (2023-10-23T13:55:49Z)
• Boost Video Frame Interpolation via Motion Adaptation [73.42573856943923]
  Video frame (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video. Existing learning-based VFI methods have achieved great success, but they still suffer from limited generalization ability. We propose a novel optimization-based VFI method that can adapt to unseen motions at test time.
  arXiv  Detail & Related papers  (2023-06-24T10:44:02Z)
• E-VFIA : Event-Based Video Frame Interpolation with Attention [8.93294761619288]
  We propose an event-based video frame with attention (E-VFIA) as a lightweight kernel-based method. E-VFIA fuses event information with standard video frames by deformable convolutions to generate high quality interpolated frames. The proposed method represents events with high temporal resolution and uses a multi-head self-attention mechanism to better encode event-based information.
  arXiv  Detail & Related papers  (2022-09-19T21:40:32Z)
• TTVFI: Learning Trajectory-Aware Transformer for Video Frame Interpolation [50.49396123016185]
  Video frame (VFI) aims to synthesize an intermediate frame between two consecutive frames. We propose a novel Trajectory-aware Transformer for Video Frame Interpolation (TTVFI) Our method outperforms other state-of-the-art methods in four widely-used VFI benchmarks.
  arXiv  Detail & Related papers  (2022-07-19T03:37:49Z)
• Exploring Motion Ambiguity and Alignment for High-Quality Video Frame Interpolation [46.02120172459727]
  We propose to relax the requirement of reconstructing an intermediate frame as close to the ground-truth (GT) as possible. We develop a texture consistency loss (TCL) upon the assumption that the interpolated content should maintain similar structures with their counterparts in the given frames.
  arXiv  Detail & Related papers  (2022-03-19T10:37:06Z)
• Asymmetric Bilateral Motion Estimation for Video Frame Interpolation [50.44508853885882]
  We propose a novel video frame algorithm based on asymmetric bilateral motion estimation (ABME) We predict symmetric bilateral motion fields to interpolate an anchor frame. We estimate asymmetric bilateral motions fields from the anchor frame to the input frames. Third, we use the asymmetric fields to warp the input frames backward and reconstruct the intermediate frame.
  arXiv  Detail & Related papers  (2021-08-15T21:11:35Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)
• SF-Net: Single-Frame Supervision for Temporal Action Localization [60.202516362976645]
  Single-frame supervision introduces extra temporal action signals while maintaining low annotation overhead. We propose a unified system called SF-Net to make use of such single-frame supervision. SF-Net significantly improves upon state-of-the-art weakly-supervised methods in terms of both segment localization and single-frame localization.
  arXiv  Detail & Related papers  (2020-03-15T15:06:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.