Mini-ResEmoteNet: Leveraging Knowledge Distillation for Human-Centered Design

• URL: http://arxiv.org/abs/2501.18538v1
• Date: Thu, 30 Jan 2025 18:06:44 GMT
• Title: Mini-ResEmoteNet: Leveraging Knowledge Distillation for Human-Centered Design
• Authors: Amna Murtada, Omnia Abdelrhman, Tahani Abdalla Attia,
• Abstract summary: Mini-ResEmoteNet models are lightweight student models tailored for usability testing. Development involves reducing the number of feature channels in each layer of the teacher model by approximately 50%, 75%, and 87.5%. Student Model A (E1) achieved a test accuracy of 76.33%, marking a 0.21% improvement over EmoNeXt.
• Score: 0.0
• License:
• Abstract: Facial Emotion Recognition has emerged as increasingly pivotal in the domain of User Experience, notably within modern usability testing, as it facilitates a deeper comprehension of user satisfaction and engagement. This study aims to extend the ResEmoteNet model by employing a knowledge distillation framework to develop Mini-ResEmoteNet models - lightweight student models - tailored for usability testing. Experiments were conducted on the FER2013 and RAF-DB datasets to assess the efficacy of three student model architectures: Student Model A, Student Model B, and Student Model C. Their development involves reducing the number of feature channels in each layer of the teacher model by approximately 50%, 75%, and 87.5%. Demonstrating exceptional performance on the FER2013 dataset, Student Model A (E1) achieved a test accuracy of 76.33%, marking a 0.21% absolute improvement over EmoNeXt. Moreover, the results exhibit absolute improvements in terms of inference speed and memory usage during inference compared to the ResEmoteNet model. The findings indicate that the proposed methods surpass other state-of-the-art approaches.

Related papers

• Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models [80.65242356955231]
  We propose a novel inference-aware fine-tuning paradigm, in which the model is fine-tuned in a manner that directly optimize the performance of the inference-time strategy. We devise the first imitation learning and reinforcement learning(RL) methods for BoN-aware fine-tuning, overcoming the challenging, non-differentiable argmax operator within BoN. Our experiments demonstrate the effectiveness of BoN-aware fine-tuning in terms of improved performance and inference-time compute.
  arXiv  Detail & Related papers  (2024-12-18T20:43:47Z)
• Improvement in Facial Emotion Recognition using Synthetic Data Generated by Diffusion Model [2.205257684291835]
  Facial Emotion Recognition (FER) plays a crucial role in computer vision, with significant applications in human-computer interaction, affective computing, and areas such as mental health monitoring and personalized learning environments. A major challenge in FER task is the class imbalance commonly found in available datasets, which can hinder both model performance and generalization. We tackle the issue of data imbalance by incorporating synthetic data augmentation and leveraging the ResEmoteNet model to enhance the overall performance on facial emotion recognition task.
  arXiv  Detail & Related papers  (2024-11-16T19:01:50Z)
• The Surprising Effectiveness of Test-Time Training for Abstract Reasoning [64.36534512742736]
  We investigate the effectiveness of test-time training (TTT) as a mechanism for improving models' reasoning capabilities. TTT significantly improves performance on ARC tasks, achieving up to 6x improvement in accuracy compared to base fine-tuned models. Our findings suggest that explicit symbolic search is not the only path to improved abstract reasoning in neural language models.
  arXiv  Detail & Related papers  (2024-11-11T18:59:45Z)
• CLDA: Collaborative Learning for Enhanced Unsupervised Domain Adaptation [15.97351561456467]
  Collaborative Learning is a method that updates the teacher's non-salient parameters using the student model and at the same time enhance the student's performance. CLDA achieves an improvement of +0.7% mIoU for teacher and +1.4% mIoU for student compared to the baseline model in the GTA to Cityscapes.
  arXiv  Detail & Related papers  (2024-09-04T13:35:15Z)
• PaCKD: Pattern-Clustered Knowledge Distillation for Compressing Memory Access Prediction Models [2.404163279345609]
  PaCKD is a pattern-Clustered Knowledge Distillation approach to compress MAP models. PaCKD yields an 8.70% higher result compared to student models trained with standard knowledge distillation and an 8.88% higher result compared to student models trained without any form of knowledge distillation.
  arXiv  Detail & Related papers  (2024-02-21T00:24:34Z)
• Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments [4.541309099803903]
  This study proposes a method for knowledge distillation (KD) of fine-tuned Large Language Models (LLMs) We specifically target the challenge of deploying these models on resource-constrained devices.
  arXiv  Detail & Related papers  (2023-12-26T01:24:25Z)
• E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning [55.50908600818483]
  Fine-tuning large-scale pretrained vision models for new tasks has become increasingly parameter-intensive. We propose an Effective and Efficient Visual Prompt Tuning (E2VPT) approach for large-scale transformer-based model adaptation. Our approach outperforms several state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2023-07-25T19:03:21Z)
• Learning Diversified Feature Representations for Facial Expression Recognition in the Wild [97.14064057840089]
  We propose a mechanism to diversify the features extracted by CNN layers of state-of-the-art facial expression recognition architectures. Experimental results on three well-known facial expression recognition in-the-wild datasets, AffectNet, FER+, and RAF-DB, show the effectiveness of our method.
  arXiv  Detail & Related papers  (2022-10-17T19:25:28Z)
• Design Amortization for Bayesian Optimal Experimental Design [70.13948372218849]
  We build off of successful variational approaches, which optimize a parameterized variational model with respect to bounds on the expected information gain (EIG) We present a novel neural architecture that allows experimenters to optimize a single variational model that can estimate the EIG for potentially infinitely many designs.
  arXiv  Detail & Related papers  (2022-10-07T02:12:34Z)
• Knowledge distillation: A good teacher is patient and consistent [71.14922743774864]
  There is a growing discrepancy in computer vision between large-scale models that achieve state-of-the-art performance and models that are affordable in practical applications. We identify certain implicit design choices, which may drastically affect the effectiveness of distillation. We obtain a state-of-the-art ResNet-50 model for ImageNet, which achieves 82.8% top-1 accuracy.
  arXiv  Detail & Related papers  (2021-06-09T17:20:40Z)
• Online Knowledge Distillation via Multi-branch Diversity Enhancement [15.523646047674717]
  We propose a new distillation method to enhance the diversity among multiple student models. We use Feature Fusion Module (FFM), which improves the performance of the attention mechanism in the network. We also use Diversification(CD) loss function to strengthen the differences between the student models.
  arXiv  Detail & Related papers  (2020-10-02T05:52:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.