Parameter Aware Mamba Model for Multi-task Dense Prediction

• URL: http://arxiv.org/abs/2511.14503v1
• Date: Tue, 18 Nov 2025 13:48:00 GMT
• Title: Parameter Aware Mamba Model for Multi-task Dense Prediction
• Authors: Xinzhuo Yu, Yunzhi Zhuge, Sitong Gong, Lu Zhang, Pingping Zhang, Huchuan Lu,
• Abstract summary: We introduce a novel decoder-based framework, Aware Mamba Model (PAMM), specifically designed for dense prediction in multi-task learning setting.<n>It features dual state space parameter experts that integrate and set task-specific parameter priors, capturing the intrinsic properties of each task.<n>We employ the Multi-Directional Hilbert Scanning method to construct multi-angle feature sequences, thereby enhancing the sequence model's perceptual capabilities for 2D data.
• Score: 69.94454603308196
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the inter-relations and interactions between tasks is crucial for multi-task dense prediction. Existing methods predominantly utilize convolutional layers and attention mechanisms to explore task-level interactions. In this work, we introduce a novel decoder-based framework, Parameter Aware Mamba Model (PAMM), specifically designed for dense prediction in multi-task learning setting. Distinct from approaches that employ Transformers to model holistic task relationships, PAMM leverages the rich, scalable parameters of state space models to enhance task interconnectivity. It features dual state space parameter experts that integrate and set task-specific parameter priors, capturing the intrinsic properties of each task. This approach not only facilitates precise multi-task interactions but also allows for the global integration of task priors through the structured state space sequence model (S4). Furthermore, we employ the Multi-Directional Hilbert Scanning method to construct multi-angle feature sequences, thereby enhancing the sequence model's perceptual capabilities for 2D data. Extensive experiments on the NYUD-v2 and PASCAL-Context benchmarks demonstrate the effectiveness of our proposed method. Our code is available at https://github.com/CQC-gogopro/PAMM.

Related papers

• Model Merging in the Essential Subspace [78.5390284258307]
  Model merging aims to integrate multiple task-specific fine-tuned models into a single multi-task model without additional training.<n>Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models.<n>We propose ESM (Essential Subspace Merging), a robust framework for effective model merging.
  arXiv  Detail & Related papers  (2026-02-23T00:33:38Z)
• Enhancing Mamba Decoder with Bidirectional Interaction in Multi-Task Dense Prediction [37.625609555296364]
  Cross-task interaction is crucial for success in multi-task dense prediction.<n>Existing methods face the trade-off between interaction completeness and computational efficiency.<n>This work proposes a Bidirectional Interaction Mamba, which incorporates novel scanning mechanisms.
  arXiv  Detail & Related papers  (2025-08-28T02:50:19Z)
• Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models [20.741460682103863]
  Sens-Merging is a sensitivity-guided coefficient adjustment method for model merging.<n>We show that Sens-Merging significantly improves performance across general knowledge, mathematical reasoning, and code generation tasks.<n>Our findings reveal important trade-offs between task-specific and cross-task scalings, providing insights for future model merging strategies.
  arXiv  Detail & Related papers  (2025-02-18T01:41:13Z)
• Pilot: Building the Federated Multimodal Instruction Tuning Framework [79.56362403673354]
  Our framework integrates two stages of "adapter on adapter" into the connector of the vision encoder and the LLM.<n>In stage 1, we extract task-specific features and client-specific features from visual information.<n>In stage 2, we build the cross-task Mixture-of-Adapters(CT-MoA) module to perform cross-task interaction.
  arXiv  Detail & Related papers  (2025-01-23T07:49:24Z)
• Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging [75.93960998357812]
  Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their capabilities across different tasks and domains.<n>Current model merging techniques focus on merging all available models simultaneously, with weight matrices-based methods being the predominant approaches.<n>We propose a training-free projection-based continual merging method that processes models sequentially.
  arXiv  Detail & Related papers  (2025-01-16T13:17:24Z)
• All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation [28.62276713652864]
  We propose Multi-Path Aggregation (MPA) integrated into existing coding models for joint human-machine vision. MPA employs a predictor to allocate latent features among task-specific paths. MPA achieves performance comparable to state-of-the-art methods in both task-specific and multi-objective optimization.
  arXiv  Detail & Related papers  (2024-09-29T11:14:21Z)
• Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports [53.637837706712794]
  We propose a Unified Trajectory Generation model, UniTraj, that processes arbitrary trajectories as masked inputs.<n>Specifically, we introduce a Ghost Spatial Masking (GSM) module, embedded within a Transformer encoder, for spatial feature extraction.<n>We benchmark three practical sports datasets, Basketball-U, Football-U, and Soccer-U, for evaluation.
  arXiv  Detail & Related papers  (2024-05-27T22:15:23Z)
• Merging Multi-Task Models via Weight-Ensembling Mixture of Experts [64.94129594112557]
  Merging Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently. Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable. We propose to merge most of the parameters while upscaling the Transformer layers to a weight-ensembling mixture of experts (MoE) module.
  arXiv  Detail & Related papers  (2024-02-01T08:58:57Z)
• Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion [86.6191592951269]
  Merging models fine-tuned from common extensively pretrained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct a multitask model that performs well across diverse tasks. We propose the CONtinuous relaxation dis (Concrete) subspace learning method to identify a common lowdimensional subspace and utilize its shared information track interference problem without sacrificing performance.
  arXiv  Detail & Related papers  (2023-12-11T07:24:54Z)
• Prompt Guided Transformer for Multi-Task Dense Prediction [14.815576352301322]
  We introduce a lightweight task-conditional model called Prompt Guided Transformer to optimize performance and model parameters. Our approach achieves state-of-the-art results among task-conditional methods while using fewer parameters, and maintains a significant balance between performance and parameter size.
  arXiv  Detail & Related papers  (2023-07-28T07:25:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.