FE-Adapter: Adapting Image-based Emotion Classifiers to Videos

• URL: http://arxiv.org/abs/2408.02421v1
• Date: Mon, 5 Aug 2024 12:27:28 GMT
• Title: FE-Adapter: Adapting Image-based Emotion Classifiers to Videos
• Authors: Shreyank N Gowda, Boyan Gao, David A. Clifton,
• Abstract summary: We present the Facial-Emotion Adapter (FE-Adapter), designed for efficient fine-tuning in video tasks. FE-Adapter can match or even surpass existing fine-tuning and video emotion models in both performance and efficiency.
• Score: 21.294212686294568
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Utilizing large pre-trained models for specific tasks has yielded impressive results. However, fully fine-tuning these increasingly large models is becoming prohibitively resource-intensive. This has led to a focus on more parameter-efficient transfer learning, primarily within the same modality. But this approach has limitations, particularly in video understanding where suitable pre-trained models are less common. Addressing this, our study introduces a novel cross-modality transfer learning approach from images to videos, which we call parameter-efficient image-to-video transfer learning. We present the Facial-Emotion Adapter (FE-Adapter), designed for efficient fine-tuning in video tasks. This adapter allows pre-trained image models, which traditionally lack temporal processing capabilities, to analyze dynamic video content efficiently. Notably, it uses about 15 times fewer parameters than previous methods, while improving accuracy. Our experiments in video emotion recognition demonstrate that the FE-Adapter can match or even surpass existing fine-tuning and video emotion models in both performance and efficiency. This breakthrough highlights the potential for cross-modality approaches in enhancing the capabilities of AI models, particularly in fields like video emotion analysis where the demand for efficiency and accuracy is constantly rising.

Related papers

• AM Flow: Adapters for Temporal Processing in Action Recognition [6.67921694218089]
  "textitAttention Map (AM) Flow" is a method for identifying pixels relevant to motion in each input video frame. AM flow allows the separation spatial and temporal processing, while providing improved results over combined-temporal processing. We present experiments on Kinetics-400, Something-Something v2, and Toyota Smarthome datasets, showcasing state-of-the-art or comparable results.
  arXiv  Detail & Related papers  (2024-11-04T13:07:22Z)
• TDS-CLIP: Temporal Difference Side Network for Image-to-Video Transfer Learning [6.329214318116305]
  We propose a memory-efficient Temporal Difference Side Network ( TDS-CLIP) to balance knowledge transferring and temporal modeling. Specifically, we introduce a Temporal Difference Adapter (TD-Adapter), which can effectively capture local temporal differences in motion features. We also designed a Side Motion Enhancement Adapter (SME-Adapter) to guide the proposed side network in efficiently learning the rich motion information in videos.
  arXiv  Detail & Related papers  (2024-08-20T09:40:08Z)
• AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction [88.70116693750452]
  Text-guided video prediction (TVP) involves predicting the motion of future frames from the initial frame according to an instruction. Previous TVP methods make significant breakthroughs by adapting Stable Diffusion for this task. We introduce the Multi-Modal Large Language Model (MLLM) to predict future video states based on initial frames and text instructions.
  arXiv  Detail & Related papers  (2024-06-10T17:02:08Z)
• ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Video [15.952896909797728]
  Adapting image models to the video domain has emerged as an efficient paradigm for solving video recognition tasks. Recent research is shifting its focus toward parameter-efficient image-to-video adaptation. We present a new adaptation paradigm (ZeroI2V) to transfer the image transformers to video recognition tasks.
  arXiv  Detail & Related papers  (2023-10-02T16:41:20Z)
• Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer [79.20605034378187]
  Video-language pre-trained models have shown remarkable success in guiding video question-answering tasks. Due to the length of video sequences, training large-scale video-based models incurs considerably higher costs than training image-based ones. This motivates us to leverage the knowledge from image-based pretraining, despite the obvious gaps between image and video domains.
  arXiv  Detail & Related papers  (2023-08-16T15:00:50Z)
• Boost Video Frame Interpolation via Motion Adaptation [73.42573856943923]
  Video frame (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video. Existing learning-based VFI methods have achieved great success, but they still suffer from limited generalization ability. We propose a novel optimization-based VFI method that can adapt to unseen motions at test time.
  arXiv  Detail & Related papers  (2023-06-24T10:44:02Z)
• AIM: Adapting Image Models for Efficient Video Action Recognition [22.805026175928997]
  We propose a method to Adapt pre-trained Image Models (AIM) for efficient video understanding. By freezing the pre-trained video model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal adaptation and joint adaptation. We show that our proposed AIM can achieve competitive or even better performance than prior arts with substantially fewer tunable parameters.
  arXiv  Detail & Related papers  (2023-02-06T18:59:17Z)
• Parameter-Efficient Image-to-Video Transfer Learning [66.82811235484607]
  Large pre-trained models for various downstream tasks of interest have recently emerged with promising performance. Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes costly in terms of model training and storage. We propose a new Spatio-Adapter for parameter-efficient fine-tuning per video task.
  arXiv  Detail & Related papers  (2022-06-27T18:02:29Z)
• Long-Short Temporal Contrastive Learning of Video Transformers [62.71874976426988]
  Self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results on par or better than those obtained with supervised pretraining on large-scale image datasets. Our approach, named Long-Short Temporal Contrastive Learning, enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
  arXiv  Detail & Related papers  (2021-06-17T02:30:26Z)
• Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning [88.71867887257274]
  We show that spatial augmentations such as cropping work well for videos too, but that previous implementations could not do this at a scale sufficient for it to work well. To address this issue, we first introduce Feature Crop, a method to simulate such augmentations much more efficiently directly in feature space. Second, we show that as opposed to naive average pooling, the use of transformer-based attention performance improves significantly.
  arXiv  Detail & Related papers  (2021-03-18T12:32:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.