DLM-VMTL:A Double Layer Mapper for heterogeneous data video Multi-task prompt learning

• URL: http://arxiv.org/abs/2408.16195v1
• Date: Thu, 29 Aug 2024 01:25:36 GMT
• Title: DLM-VMTL:A Double Layer Mapper for heterogeneous data video Multi-task prompt learning
• Authors: Zeyi Bo, Wuxi Sun, Ye Jin,
• Abstract summary: Multi-Task Learning makes the visual task acquire the rich shareable knowledge from other tasks while joint training. Heterogenous data video multi-task prompt learning (VMTL) method is proposed to address above problem. Double-Layers Mapper(DLM) is proposed to extract the shareable knowledge into visual promptS and align it with representation of primary task.
• Score: 2.4121373594852846
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In recent years, the parameters of backbones of Video Understanding tasks continue to increase and even reach billion-level. Whether fine-tuning a specific task on the Video Foundation Model or pre-training the model designed for the specific task, incurs a lot of overhead. How to make these models play other values than their own tasks becomes a worthy question. Multi-Task Learning(MTL) makes the visual task acquire the rich shareable knowledge from other tasks while joint training. It is fully explored in Image Recognition tasks especially dense predict tasks. Nevertheless, it is rarely used in video domain due to the lack of multi-labels video data. In this paper, a heterogenous data video multi-task prompt learning (VMTL) method is proposed to address above problem. It's different from it in image domain, a Double-Layers Mapper(DLM) is proposed to extract the shareable knowledge into visual promptS and align it with representation of primary task. Extensive experiments prove that our DLM-VMTL performs better than baselines on 6 different video understanding tasks and 11 datasets.

Related papers

• VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks [89.24440488456405]
  VisionLLM v2 is an end-to-end generalist multimodal large model (MLLM) It unifies visual perception, understanding, and generation within a single framework.
  arXiv  Detail & Related papers  (2024-06-12T16:44:50Z)
• LiPost: Improved Content Understanding With Effective Use of Multi-task Contrastive Learning [2.611731148829789]
  We fine-tune a pre-trained, transformer-based LLM using multi-task contrastive learning with data from a diverse set of semantic labeling tasks. Our model outperforms the baseline on zero shot learning and offers improved multilingual support. This work provides a robust foundation for vertical teams across LinkedIn to customize and fine-tune the LLM to their specific applications.
  arXiv  Detail & Related papers  (2024-05-18T17:28:29Z)
• TransPrompt v2: A Transferable Prompting Framework for Cross-task Text Classification [37.824031151922604]
  We propose TransPrompt v2, a novel transferable prompting framework for few-shot learning across similar or distant text classification tasks. For learning across similar tasks, we employ a multi-task meta-knowledge acquisition (MMA) procedure to train a meta-learner. For learning across distant tasks, we inject the task type descriptions into the prompt, and capture the intra-type and inter-type prompt embeddings.
  arXiv  Detail & Related papers  (2023-08-29T04:16:57Z)
• FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks [129.49630356651454]
  We propose a novel FAshion-focused Multi-task Efficient learning method for Vision-and-Language tasks (FAME-ViL) Our FAME-ViL can save 61.5% of parameters over alternatives, while significantly outperforming the conventional independently trained single-task models.
  arXiv  Detail & Related papers  (2023-03-04T19:07:48Z)
• Multimodal Subtask Graph Generation from Instructional Videos [51.96856868195961]
  Real-world tasks consist of multiple inter-dependent subtasks. In this work, we aim to model the causal dependencies between such subtasks from instructional videos describing the task. We present Multimodal Subtask Graph Generation (MSG2), an approach that constructs a Subtask Graph defining the dependency between a task's subtasks relevant to a task from noisy web videos.
  arXiv  Detail & Related papers  (2023-02-17T03:41:38Z)
• MINOTAUR: Multi-task Video Grounding From Multimodal Queries [70.08973664126873]
  We present a single, unified model for tackling query-based video understanding in long-form videos. In particular, our model can address all three tasks of the Ego4D Episodic Memory benchmark.
  arXiv  Detail & Related papers  (2023-02-16T04:00:03Z)
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation [124.02278735049235]
  VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. We evaluate various baseline methods with and without large-scale VidL pre-training. The significant gap between our best model and human performance calls for future study for advanced VidL models.
  arXiv  Detail & Related papers  (2021-06-08T18:34:21Z)
• Understanding and Improving Information Transfer in Multi-Task Learning [14.43111978531182]
  We study an architecture with a shared module for all tasks and a separate output module for each task. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning.
  arXiv  Detail & Related papers  (2020-05-02T23:43:52Z)
• MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning [82.62433731378455]
  We show that tasks with high affinity at a certain scale are not guaranteed to retain this behaviour at other scales. We propose a novel architecture, namely MTI-Net, that builds upon this finding.
  arXiv  Detail & Related papers  (2020-01-19T21:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.