Efficient Multi-Task Inferencing with a Shared Backbone and Lightweight Task-Specific Adapters for Automatic Scoring

• URL: http://arxiv.org/abs/2412.21065v1
• Date: Mon, 30 Dec 2024 16:34:11 GMT
• Title: Efficient Multi-Task Inferencing with a Shared Backbone and Lightweight Task-Specific Adapters for Automatic Scoring
• Authors: Ehsan Latif, Xiaoming Zhai,
• Abstract summary: This paper proposes a shared backbone model architecture enhanced with lightweight LoRA adapters for task-specific fine-tuning. It targets the automated scoring of student responses across 27 mutually exclusive tasks.
• Score: 1.2556373621040728
• License:
• Abstract: The integration of Artificial Intelligence (AI) in education requires scalable and efficient frameworks that balance performance, adaptability, and cost. This paper addresses these needs by proposing a shared backbone model architecture enhanced with lightweight LoRA adapters for task-specific fine-tuning, targeting the automated scoring of student responses across 27 mutually exclusive tasks. By achieving competitive performance (average QWK of 0.848 compared to 0.888 for fully fine-tuned models) while reducing GPU memory consumption by 60% and inference latency by 40%, the framework demonstrates significant efficiency gains. This approach aligns with the workshops' focus on improving language models for educational tasks, creating responsible innovations for cost-sensitive deployment, and supporting educators by streamlining assessment workflows. The findings underscore the potential of scalable AI to enhance learning outcomes while maintaining fairness and transparency in automated scoring systems.

Related papers

• Satisfaction-Aware Incentive Scheme for Federated Learning in Industrial Metaverse: DRL-Based Stackbelberg Game Approach [52.79672516203574]
  Industrial Metaverse uses federated learning and meta-computing to train models in a distributed manner while ensuring data privacy. This paper designs a satisfaction function that accounts for data size, Age of Information (AoI), and training latency. The satisfaction function is incorporated into the utility functions to incentivize node participation in model training.
  arXiv  Detail & Related papers  (2025-02-10T03:33:36Z)
• Heterogeneity-aware Personalized Federated Learning via Adaptive Dual-Agent Reinforcement Learning [15.61141633436468]
  Federated Learning (FL) empowers multiple clients to collaboratively train machine learning models without sharing local data. This paper proposes a novel Heterogeneity-aware Personalized Federated Learning method, named HAPFL, via multi-level Reinforcement Learning (RL) mechanisms. Experimental results across multiple benchmark datasets demonstrate that HAPFL not only achieves high accuracy but also substantially reduces the overall training time by 20.9%-40.4%.
  arXiv  Detail & Related papers  (2025-01-28T14:08:57Z)
• Enhancing IoT Network Security through Adaptive Curriculum Learning and XAI [0.3958317527488535]
  This study presents a scalable and lightweight curriculum learning framework enhanced with Explainable AI (XAI) techniques, including LIME. The proposed model employs novel neural network architecture utilized at every stage of Curriculum Learning to efficiently capture and focus on both short- and long-term temporal dependencies. Experimental results demonstrate 98% accuracy on CIC-IoV-2024 and CIC-APT-IIoT-2024 datasets and 97% on EDGE-IIoT.
  arXiv  Detail & Related papers  (2025-01-20T17:32:01Z)
• Semantic Knowledge Distillation for Onboard Satellite Earth Observation Image Classification [28.08042498882207]
  This study presents an innovative dynamic weighting knowledge distillation (KD) framework tailored for efficient Earth observation (EO) image classification (IC) in resource-constrained settings. Our framework enables lightweight student models to surpass 90% in accuracy, precision, and recall, adhering to the stringent confidence thresholds necessary for reliable classification tasks. Remarkably, ResNet8 delivers substantial efficiency gains, achieving a 97.5% reduction in parameters, a 96.7% decrease in FLOPs, an 86.2% cut in power consumption, and a 63.5% increase in inference speed over MobileViT.
  arXiv  Detail & Related papers  (2024-10-31T21:13:40Z)
• CodeACT: Code Adaptive Compute-efficient Tuning Framework for Code LLMs [30.441431693349866]
  Existing methods generate vast amounts of synthetic data for fine-tuning, leading to inefficiencies in training. CodeACT introduces the Complexity and Diversity Aware Sampling (CDAS) method to select high-quality training data. CodeACT achieves an 8.6% performance increase on HumanEval, reduces training time by 78%, and decreases peak GPU memory usage by 27%.
  arXiv  Detail & Related papers  (2024-08-05T02:38:48Z)
• GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment [74.40196814292426]
  We introduce a novel and intuitive Guidance-based Knowledge Transfer (GKT) framework. GKT uses a larger Large Language Models as a ''teacher'' to create guidance prompts, paired with a smaller ''student'' model to finalize responses. It achieves a maximum accuracy improvement of 14.18%, along with a 10.72 times speed-up on GSM8K and an accuracy improvement of 14.00 % along with a 7.73 times speed-up in CSQA.
  arXiv  Detail & Related papers  (2024-05-30T02:37:35Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation [56.053397775016755]
  We propose a sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students.
  arXiv  Detail & Related papers  (2023-08-17T17:17:08Z)
• GAT: Guided Adversarial Training with Pareto-optimal Auxiliary Tasks [73.88590165742721]
  We propose a novel adversarial training technique that exploits auxiliary tasks under a limited set of training data. Our approach extends single-task models into multi-task models during the min-max optimization of adversarial training. We demonstrate that guided multi-task learning is an actionable and promising avenue to push further the boundaries of model robustness.
  arXiv  Detail & Related papers  (2023-02-06T16:23:24Z)
• Composite Learning for Robust and Effective Dense Predictions [81.2055761433725]
  Multi-task learning promises better model generalization on a target task by jointly optimizing it with an auxiliary task. We find that jointly training a dense prediction (target) task with a self-supervised (auxiliary) task can consistently improve the performance of the target task, while eliminating the need for labeling auxiliary tasks.
  arXiv  Detail & Related papers  (2022-10-13T17:59:16Z)
• AttentionLite: Towards Efficient Self-Attention Models for Vision [9.957033392865982]
  We propose a novel framework for producing a class of parameter and compute efficient models called AttentionLitesuitable for resource-constrained applications. We can simultaneously distill knowledge from a compute-heavy teacher while also pruning the student model in a single pass of training.
  arXiv  Detail & Related papers  (2020-12-21T17:54:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.