Prototypical Transformer as Unified Motion Learners

• URL: http://arxiv.org/abs/2406.01559v1
• Date: Mon, 3 Jun 2024 17:41:28 GMT
• Title: Prototypical Transformer as Unified Motion Learners
• Authors: Cheng Han, Yawen Lu, Guohao Sun, James C. Liang, Zhiwen Cao, Qifan Wang, Qiang Guan, Sohail A. Dianat, Raghuveer M. Rao, Tong Geng, Zhiqiang Tao, Dongfang Liu,
• Abstract summary: Prototypical Transformer (ProtoFormer) is a framework that approaches various motion tasks from a prototype perspective. ProtoFormer seamlessly integrates prototype learning with Transformer by thoughtfully considering motion dynamics.
• Score: 38.31482767855841
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we introduce the Prototypical Transformer (ProtoFormer), a general and unified framework that approaches various motion tasks from a prototype perspective. ProtoFormer seamlessly integrates prototype learning with Transformer by thoughtfully considering motion dynamics, introducing two innovative designs. First, Cross-Attention Prototyping discovers prototypes based on signature motion patterns, providing transparency in understanding motion scenes. Second, Latent Synchronization guides feature representation learning via prototypes, effectively mitigating the problem of motion uncertainty. Empirical results demonstrate that our approach achieves competitive performance on popular motion tasks such as optical flow and scene depth. Furthermore, it exhibits generality across various downstream tasks, including object tracking and video stabilization.

Related papers

• ProMotion: Prototypes As Motion Learners [46.08051377180652]
  We introduce ProMotion, a unified prototypical framework engineered to model fundamental motion tasks. ProMotion offers a range of compelling attributes that set it apart from current task-specific paradigms. We capitalize on a dual mechanism involving the feature denoiser and the prototypical learner to decipher the intricacies of motion.
  arXiv  Detail & Related papers  (2024-06-07T15:10:33Z)
• Motion Inversion for Video Customization [31.607669029754874]
  We present a novel approach for motion in generation, addressing the widespread gap in the exploration of motion representation within video models. We introduce Motion Embeddings, a set of explicit, temporally coherent embeddings derived from given video. Our contributions include a tailored motion embedding for customization tasks and a demonstration of the practical advantages and effectiveness of our method.
  arXiv  Detail & Related papers  (2024-03-29T14:14:22Z)
• Motion Flow Matching for Human Motion Synthesis and Editing [75.13665467944314]
  We propose emphMotion Flow Matching, a novel generative model for human motion generation featuring efficient sampling and effectiveness in motion editing applications. Our method reduces the sampling complexity from thousand steps in previous diffusion models to just ten steps, while achieving comparable performance in text-to-motion and action-to-motion generation benchmarks.
  arXiv  Detail & Related papers  (2023-12-14T12:57:35Z)
• UniST: Towards Unifying Saliency Transformer for Video Saliency Prediction and Detection [9.063895463649414]
  We introduce the Unified Saliency Transformer (UniST) framework, which comprehensively utilizes the essential attributes of video saliency prediction and video salient object detection. To the best of our knowledge, this is the first work that explores designing a transformer structure for both saliency modeling tasks.
  arXiv  Detail & Related papers  (2023-09-15T07:39:53Z)
• MotionTrack: Learning Motion Predictor for Multiple Object Tracking [68.68339102749358]
  We introduce a novel motion-based tracker, MotionTrack, centered around a learnable motion predictor. Our experimental results demonstrate that MotionTrack yields state-of-the-art performance on datasets such as Dancetrack and SportsMOT.
  arXiv  Detail & Related papers  (2023-06-05T04:24:11Z)
• Human MotionFormer: Transferring Human Motions with Vision Transformers [73.48118882676276]
  Human motion transfer aims to transfer motions from a target dynamic person to a source static one for motion synthesis. We propose Human MotionFormer, a hierarchical ViT framework that leverages global and local perceptions to capture large and subtle motion matching. Experiments show that our Human MotionFormer sets the new state-of-the-art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2023-02-22T11:42:44Z)
• Motion-Attentive Transition for Zero-Shot Video Object Segmentation [99.44383412488703]
  We present a Motion-Attentive Transition Network (MATNet) for zero-shot object segmentation. An asymmetric attention block, called Motion-Attentive Transition (MAT), is designed within a two-stream encoder. In this way, the encoder becomes deeply internative, allowing for closely hierarchical interactions between object motion and appearance.
  arXiv  Detail & Related papers  (2020-03-09T16:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.