Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction

• URL: http://arxiv.org/abs/2608.28078v1
• Date: Fri, 28 Aug 2026 08:48:22 GMT
• Title: Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction
• Abstract summary: We propose a multi-task dense prediction framework that estimates a compact task state from global visual context.<n>MemMTL refines it through a learnable task-state prototype memory.<n>A separate task-agnostic residual bank provides a common adaptation path.
• Score: 65.90697836568579
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision foundation backbones provide strong representations for dense prediction, yet a single shared feature still needs to support tasks with different, image-dependent adaptation requirements. We propose MemMTL, a multi-task dense prediction framework that estimates a compact task state from global visual context and refines it through a learnable task-state prototype memory. The refined state is converted into task-conditioned expert logits and combined with token-level logits before sparse top-$k$ selection over a local expert bank shared by all tasks. A separate task-agnostic residual bank provides a common adaptation path, and both paths are added once to the backbone feature before task-specific prediction. We specify a matched evaluation protocol on NYUD-v2 and PASCAL-Context with SAM 3 and ViT-L backbones to measure predictive quality, computational cost, and the contributions of task-state conditioning, prototype retrieval, and sparse routing. The numerical record in the present working draft predates this canonical implementation and must be regenerated before it can support empirical claims.

Related papers

• Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models [10.94594187411586]
  Hierarchical Skill Retrieval (HSR) is a retrieval framework for data-efficient VLA adaptation.<n>We show HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline.
  arXiv  Detail & Related papers  (2026-08-25T04:04:49Z)
• IntHQ: Task-Interactive Hierarchical Query on Dual-Stream Representations for Generative Recommendation [20.546679348797905]
  We show that existing multi-taskers are inherently prone to a threefold collapse.<n>We propose InHQ, a multi-task generative recommender with three components each alleviating one collapse.<n>In offline evaluations, InHQ consistently outperforms competitive backbones under four representative task-head configurations.
  arXiv  Detail & Related papers  (2026-08-10T14:13:47Z)
• A Foundation Model for Instruction-Conditioned In-Context Time Series Tasks [0.7677431882779153]
  In-context learning (ICL) allows a model to adapt at inference time by conditioning on examples rather than updating parameters.<n>We present a foundation model for instruction-conditioned in-context time-series tasks based on a quantile-regression T5 encoder-decoder.
  arXiv  Detail & Related papers  (2026-03-23T21:24:41Z)
• PanMatch: Unleashing the Potential of Large Vision Models for Unified Matching Models [80.65273820998875]
  We present PanMatch, a versatile foundation model for robust correspondence matching.<n>Our key insight is that any two-frame correspondence matching task can be addressed within a 2D displacement estimation framework.<n>PanMatch achieves multi-task integration by endowing displacement estimation algorithms with unprecedented generalization capabilities.
  arXiv  Detail & Related papers  (2025-07-11T08:18:52Z)
• Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions [44.78165979575075]
  We propose a novel approach to optimize a set of compact learnable hierarchical task tokens.<n>The global task tokens are designed for effective cross-task feature interactions in a global context.<n>A group of fine-grained task-specific spatial tokens for each task is learned from the corresponding global task tokens.<n>The learned global and local fine-grained task tokens are further used to discover pseudo task-specific dense labels at different levels of granularity.
  arXiv  Detail & Related papers  (2024-11-27T23:53:27Z)
• A Dynamic Feature Interaction Framework for Multi-task Visual Perception [100.98434079696268]
  We devise an efficient unified framework to solve multiple common perception tasks. These tasks include instance segmentation, semantic segmentation, monocular 3D detection, and depth estimation. Our proposed framework, termed D2BNet, demonstrates a unique approach to parameter-efficient predictions for multi-task perception.
  arXiv  Detail & Related papers  (2023-06-08T09:24:46Z)
• Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsing [110.4684789199555]
  We introduce scenario-based semantic parsing: a variant of the original task which first requires disambiguating an utterance's "scenario" This formulation enables us to isolate coarse-grained and fine-grained aspects of the task, each of which we solve with off-the-shelf neural modules. Our model is modular, differentiable, interpretable, and allows us to garner extra supervision from scenarios.
  arXiv  Detail & Related papers  (2022-02-02T08:00:21Z)
• Exploring Relational Context for Multi-Task Dense Prediction [76.86090370115]
  We consider a multi-task environment for dense prediction tasks, represented by a common backbone and independent task-specific heads. We explore various attention-based contexts, such as global and local, in the multi-task setting. We propose an Adaptive Task-Relational Context module, which samples the pool of all available contexts for each task pair.
  arXiv  Detail & Related papers  (2021-04-28T16:45:56Z)
• Automated Concatenation of Embeddings for Structured Prediction [75.44925576268052]
  We propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model.
  arXiv  Detail & Related papers  (2020-10-10T14:03:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.