HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts

• URL: http://arxiv.org/abs/2608.29790v2
• Date: Tue, 01 Sep 2026 05:22:08 GMT
• Title: HiVe: Beyond Static Prompts for Multitask Learning via Hierarchy-based Vertical Mixture-of-Experts
• Abstract summary: HiVe is a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization.<n> experiments show that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.
• Score: 9.080108688436466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models (LLMs) continue to scale, parameter-efficient fine-tuning (PEFT) has become a practical alternative to full-parameter adaptation. Prompt tuning is effective, but existing approaches either use flat prompt structures or hierarchical structures with fixed prompt composition, limiting adaptive prompt specialization. To address this limitation, we propose HiVe, a prompt tuning framework that models prompts at multiple levels and enables input-dependent specialization. HiVe constructs a prompt hierarchy by leveraging inter-task relationships during training, and employs a vertical mixture-of-experts (V-MoE) mechanism at inference time to compose prompts up to the level of specialization required for each input. Experiments show that HiVe consistently outperforms strong prompt tuning baselines across diverse tasks.

Related papers

• ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning [16.21619173438734]
  Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities.<n>Recent methods often employ sparse architectures like Mixture of LoRA Experts with image-text similarity routing.<n>We propose ProtoAda, a prototype-guided adaptive tuning framework.
  arXiv  Detail & Related papers  (2026-06-01T17:59:13Z)
• Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning [13.499744113926505]
  DRAPE is a prompt-learning framework that synthesizes continuous instance-specific soft prompts for MCIT.<n>DRAPE achieves state-of-the-art performance among representative prompt-based and LoRA-based continual-learning baselines.
  arXiv  Detail & Related papers  (2026-05-11T15:59:06Z)
• Expert Pyramid Tuning: Efficient Parameter Fine-Tuning for Expertise-Driven Task Allocation [7.579163774245479]
  Expert Pyramid Tuning (EPT) is a novel architecture that integrates the multi-scale feature pyramid concept from computer vision into the realm of PEFT.<n>EPT significantly outperforms SOTA MoE-LoRA variants.
  arXiv  Detail & Related papers  (2026-03-13T02:23:41Z)
• pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation [68.3777121585281]
  We propose a novel Mixture-of-Experts prompt tuning method called pMoE.<n>The proposed pMoE significantly enhances the model's versatility and applicability to a broad spectrum of tasks.<n>We conduct extensive experiments across 47 adaptation tasks, including both classification and segmentation in general and medical domains.
  arXiv  Detail & Related papers  (2026-02-26T12:27:06Z)
• Teaching Prompts to Coordinate: Hierarchical Layer-Grouped Prompt Tuning for Continual Learning [69.17264556340244]
  We propose a novel hierarchical layer-grouped prompt tuning method for continual learning.<n>It improves model stability in two ways: (i) Layers in the same group share roughly the same prompts, which are adjusted by position encoding.<n>It utilizes a single task-specific root prompt to learn to generate sub-prompts for each layer group.
  arXiv  Detail & Related papers  (2025-11-15T08:15:51Z)
• GMoPE:A Prompt-Expert Mixture Framework for Graph Foundation Models [30.023472202549076]
  Graph Neural Networks (GNNs) have demonstrated impressive performance on task-specific benchmarks, yet their ability to generalize across diverse domains and tasks remains limited.<n>We propose GMoPE, a framework that seamlessly integrates the Mixture-of-Experts (MoE) architecture with prompt-based learning for graphs.<n>We show that GMoPE consistently outperforms state-of-the-art baselines and achieves performance comparable to full parameter fine-tuning.
  arXiv  Detail & Related papers  (2025-11-05T07:28:51Z)
• PromptFlow: Training Prompts Like Neural Networks [17.90494213352502]
  Large Language Models (LLMs) have demonstrated profound impact on Natural Language Processing (NLP) tasks.<n>Recent advances in prompt engineering offer a promising alternative to extensive retraining.<n>We propose the PromptFlow, a modular training framework inspired by meta-prompts, operators, optimization, and evaluators.
  arXiv  Detail & Related papers  (2025-10-14T07:56:12Z)
• One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning [52.966712416640085]
  We propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies.<n>SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-09-29T08:54:58Z)
• MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion [29.46189153751869]
  Mixture of Prompt Experts (MoPE) is first technique designed to overcome limitations by decomposing standard prompts.<n>Our MoPE-based fusion method exhibits greater expressiveness, scaling more effectively with the training data and the overall number of trainable parameters.
  arXiv  Detail & Related papers  (2024-03-14T17:47:10Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• Prompt-Matched Semantic Segmentation [96.99924127527002]
  The objective of this work is to explore how to effectively adapt pre-trained foundation models to various downstream tasks of image semantic segmentation. We propose a novel Inter-Stage Prompt-Matched Framework, which maintains the original structure of the foundation model while generating visual prompts adaptively for task-oriented tuning. A lightweight module termed Semantic-aware Prompt Matcher is then introduced to hierarchically interpolate between two stages to learn reasonable prompts for each specific task.
  arXiv  Detail & Related papers  (2022-08-22T09:12:53Z)
• IDPG: An Instance-Dependent Prompt Generation Method [58.45110542003139]
  Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. We propose a conditional prompt generation method to generate prompts for each input instance.
  arXiv  Detail & Related papers  (2022-04-09T15:45:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.