FreqMixFormerV2: Lightweight Frequency-aware Mixed Transformer for Human Skeleton Action Recognition

• URL: http://arxiv.org/abs/2412.20621v1
• Date: Sun, 29 Dec 2024 23:52:40 GMT
• Title: FreqMixFormerV2: Lightweight Frequency-aware Mixed Transformer for Human Skeleton Action Recognition
• Authors: Wenhan Wu, Pengfei Wang, Chen Chen, Aidong Lu,
• Abstract summary: FreqMixForemrV2 is built upon the Frequency-aware Mixed Transformer (FreqMixFormer) for identifying subtle and discriminative actions. The proposed model achieves a superior balance between efficiency and accuracy, outperforming state-of-the-art methods with only 60% of the parameters.
• Score: 9.963966059349731
• License:
• Abstract: Transformer-based human skeleton action recognition has been developed for years. However, the complexity and high parameter count demands of these models hinder their practical applications, especially in resource-constrained environments. In this work, we propose FreqMixForemrV2, which was built upon the Frequency-aware Mixed Transformer (FreqMixFormer) for identifying subtle and discriminative actions with pioneered frequency-domain analysis. We design a lightweight architecture that maintains robust performance while significantly reducing the model complexity. This is achieved through a redesigned frequency operator that optimizes high-frequency and low-frequency parameter adjustments, and a simplified frequency-aware attention module. These improvements result in a substantial reduction in model parameters, enabling efficient deployment with only a minimal sacrifice in accuracy. Comprehensive evaluations of standard datasets (NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets) demonstrate that the proposed model achieves a superior balance between efficiency and accuracy, outperforming state-of-the-art methods with only 60% of the parameters.

Related papers

• FreSh: Frequency Shifting for Accelerated Neural Representation Learning [11.175745750843484]
  Implicit Neural Representations (INRs) have recently gained attention as a powerful approach for continuously representing signals such as images, videos, and 3D shapes using multilayer perceptrons (MLPs) Low-frequency details are known to exhibit a low-frequency bias, limiting their ability to capture high-frequency details accurately. We propose frequency shifting (or FreSh) to align the frequency spectrum of the initial output with that of the target signal.
  arXiv  Detail & Related papers  (2024-10-07T14:05:57Z)
• Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization [37.35829410807451]
  This paper introduces PeriodWave-Turbo, a high-fidelity and high-efficient waveform generation model via adversarial flow matching optimization. It only requires 1,000 steps of fine-tuning to achieve state-of-the-art performance across various objective metrics. By scaling up the backbone of PeriodWave from 29M to 70M parameters for improved generalization, PeriodWave-Turbo achieves unprecedented performance.
  arXiv  Detail & Related papers  (2024-08-15T08:34:00Z)
• LORTSAR: Low-Rank Transformer for Skeleton-based Action Recognition [4.375744277719009]
  LORTSAR is applied to two leading Transformer-based models, "Hyperformer" and "STEP-CATFormer" Our method can reduce the number of model parameters substantially with negligible degradation or even performance increase in recognition accuracy. This confirms that SVD combined with post-compression fine-tuning can boost model efficiency, paving the way for more sustainable, lightweight, and high-performance technologies in human action recognition.
  arXiv  Detail & Related papers  (2024-07-19T20:19:41Z)
• Uncertainty Guided Adaptive Warping for Robust and Efficient Stereo Matching [77.133400999703]
  Correlation based stereo matching has achieved outstanding performance. Current methods with a fixed model do not work uniformly well across various datasets. This paper proposes a new perspective to dynamically calculate correlation for robust stereo matching.
  arXiv  Detail & Related papers  (2023-07-26T09:47:37Z)
• MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering [66.05768870785548]
  Finetuning pretrained Vision-Language Models (VLMs) has been a prevailing paradigm for achieving state-of-the-art performance in Visual Question Answering (VQA) Current parameter-efficient tuning methods dramatically reduce the number of tunable parameters, but there still exists a significant performance gap with full finetuning. We propose MixPHM, a redundancy-aware parameter-efficient tuning method that outperforms full finetuning in low-resource VQA.
  arXiv  Detail & Related papers  (2023-03-02T13:28:50Z)
• FAMLP: A Frequency-Aware MLP-Like Architecture For Domain Generalization [73.41395947275473]
  We propose a novel frequency-aware architecture, in which the domain-specific features are filtered out in the transformed frequency domain. Experiments on three benchmarks demonstrate significant performance, outperforming the state-of-the-art methods by a margin of 3%, 4% and 9%, respectively.
  arXiv  Detail & Related papers  (2022-03-24T07:26:29Z)
• An Efficient Deep Learning Model for Automatic Modulation Recognition Based on Parameter Estimation and Transformation [3.3941243094128035]
  This letter proposes an efficient DL-AMR model based on phase parameter estimation and transformation. Our model is more competitive in training time and test time than the benchmark models with similar recognition accuracy.
  arXiv  Detail & Related papers  (2021-10-11T03:28:28Z)
• MoEfication: Conditional Computation of Transformer Models for Efficient Inference [66.56994436947441]
  Transformer-based pre-trained language models can achieve superior performance on most NLP tasks due to large parameter capacity, but also lead to huge computation cost. We explore to accelerate large-model inference by conditional computation based on the sparse activation phenomenon. We propose to transform a large model into its mixture-of-experts (MoE) version with equal model size, namely MoEfication.
  arXiv  Detail & Related papers  (2021-10-05T02:14:38Z)
• Fourier Space Losses for Efficient Perceptual Image Super-Resolution [131.50099891772598]
  We show that it is possible to improve the performance of a recently introduced efficient generator architecture solely with the application of our proposed loss functions. We show that our losses' direct emphasis on the frequencies in Fourier-space significantly boosts the perceptual image quality. The trained generator achieves comparable results with and is 2.4x and 48x faster than state-of-the-art perceptual SR methods RankSRGAN and SRFlow respectively.
  arXiv  Detail & Related papers  (2021-06-01T20:34:52Z)
• Low-Complexity Models for Acoustic Scene Classification Based on Receptive Field Regularization and Frequency Damping [7.0349768355860895]
  We investigate and compare several well-known methods to reduce the number of parameters in neural networks. We show that we can achieve high-performing low-complexity models by applying specific restrictions on the Receptive Field. We propose a filter-damping technique for regularizing the RF of models, without altering their architecture.
  arXiv  Detail & Related papers  (2020-11-05T16:34:11Z)
• Highly Efficient Salient Object Detection with 100K Parameters [137.74898755102387]
  We propose a flexible convolutional module, namely generalized OctConv (gOctConv), to efficiently utilize both in-stage and cross-stages multi-scale features. We build an extremely light-weighted model, namely CSNet, which achieves comparable performance with about 0.2% (100k) of large models on popular object detection benchmarks.
  arXiv  Detail & Related papers  (2020-03-12T07:00:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.