AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation
- URL: http://arxiv.org/abs/2304.09790v1
- Date: Wed, 19 Apr 2023 16:18:47 GMT
- Title: AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation
- Authors: Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo,
Ming-Ming Cheng
- Abstract summary: We present All-Pairs Multi-Field Transforms (AMT), a new network architecture for video framegithub.
It is based on two essential designs. First, we build bidirectional volumes for all pairs of pixels, and use the predicted bilateral flows to retrieve correlations.
Second, we derive multiple groups of fine-grained flow fields from one pair of updated coarse flows for performing backward warping on the input frames separately.
- Score: 80.33846577924363
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: We present All-Pairs Multi-Field Transforms (AMT), a new network architecture
for video frame interpolation. It is based on two essential designs. First, we
build bidirectional correlation volumes for all pairs of pixels, and use the
predicted bilateral flows to retrieve correlations for updating both flows and
the interpolated content feature. Second, we derive multiple groups of
fine-grained flow fields from one pair of updated coarse flows for performing
backward warping on the input frames separately. Combining these two designs
enables us to generate promising task-oriented flows and reduce the
difficulties in modeling large motions and handling occluded areas during frame
interpolation. These qualities promote our model to achieve state-of-the-art
performance on various benchmarks with high efficiency. Moreover, our
convolution-based model competes favorably compared to Transformer-based models
in terms of accuracy and efficiency. Our code is available at
https://github.com/MCG-NKU/AMT.
Related papers
- Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers [13.480259378415505]
BiXT scales linearly with input size in terms of computational cost and memory consumption.
BiXT is inspired by the Perceiver architectures but replaces iterative attention with an efficient bi-directional cross-attention module.
By combining efficiency with the generality and performance of a full Transformer architecture, BiXT can process longer sequences.
arXiv Detail & Related papers (2024-02-19T13:38:15Z) - JAX-Fluids 2.0: Towards HPC for Differentiable CFD of Compressible
Two-phase Flows [0.0]
JAX-Fluids is a Python-based fully-differentiable CFD solver designed for compressible single- and two-phase flows.
We introduce a parallelization strategy utilizing JAX primitive operations that scales efficiently on GPU (up to 512 NVIDIA A100 graphics cards) and TPU (up to 1024 TPU v3 cores) HPC systems.
The new code version offers enhanced two-phase flow modeling capabilities.
arXiv Detail & Related papers (2024-02-07T19:05:27Z) - Motion-Aware Video Frame Interpolation [49.49668436390514]
We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames.
It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
arXiv Detail & Related papers (2024-02-05T11:00:14Z) - Video Frame Interpolation with Many-to-many Splatting and Spatial
Selective Refinement [83.60486465697318]
We propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently.
For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames.
We extend an M2M++ framework by introducing a flexible Spatial Selective Refinement component, which allows for trading computational efficiency for quality and vice versa.
arXiv Detail & Related papers (2023-10-29T09:09:32Z) - On Optimizing the Communication of Model Parallelism [74.15423270435949]
We study a novel and important communication pattern in large-scale model-parallel deep learning (DL)
In cross-mesh resharding, a sharded tensor needs to be sent from a source device mesh to a destination device mesh.
We propose two contributions to address cross-mesh resharding: an efficient broadcast-based communication system, and an "overlapping-friendly" pipeline schedule.
arXiv Detail & Related papers (2022-11-10T03:56:48Z) - Sparsity-guided Network Design for Frame Interpolation [39.828644638174225]
We present a compression-driven network design for frame-based algorithms.
We leverage model pruning through sparsity-inducing optimization to greatly reduce the model size.
We achieve a considerable performance gain with a quarter of the size of the original AdaCoF.
arXiv Detail & Related papers (2022-09-09T23:13:25Z) - Joint Spatial-Temporal and Appearance Modeling with Transformer for
Multiple Object Tracking [59.79252390626194]
We propose a novel solution named TransSTAM, which leverages Transformer to model both the appearance features of each object and the spatial-temporal relationships among objects.
The proposed method is evaluated on multiple public benchmarks including MOT16, MOT17, and MOT20, and it achieves a clear performance improvement in both IDF1 and HOTA.
arXiv Detail & Related papers (2022-05-31T01:19:18Z) - Spatio-Temporal Multi-Flow Network for Video Frame Interpolation [3.6053802212032995]
Video frame (VFI) is a very active research topic, with applications spanning computer vision, post production and video encoding.
We present a novel deep learning based VFI method, ST-MFNet, based on a Spatio-Temporal Multi-Flow architecture.
arXiv Detail & Related papers (2021-11-30T15:18:46Z) - DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with
Flow-Guided Attentive Correlation and Recursive Boosting [50.17500790309477]
DeMFI-Net is a joint deblurring and multi-frame framework.
It converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate.
It achieves state-of-the-art (SOTA) performances for diverse datasets.
arXiv Detail & Related papers (2021-11-19T00:00:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.