Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic

• URL: http://arxiv.org/abs/2606.20734v1
• Date: Wed, 17 Jun 2026 14:16:12 GMT
• Title: Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic
• Abstract summary: Open Vocabulary Action Recognition (OVAR) enables the recognition of novel actions by leveraging vision-language representations.<n>We propose an alternative paradigm that bypasses target-domain training and recombines knowledge from existing datasets and models.<n>We show that, in out-of-distribution settings, the resulting merged model achieves superior zero-shot generalization to the pre-trained base model.
• Score: 45.298314099319775
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open Vocabulary Action Recognition (OVAR) enables the recognition of novel actions by leveraging vision-language representations, overcoming the limitations of traditional closed-set approaches. However, achieving robust performance in real-world scenarios typically requires domain-specific fine-tuning, which is often costly and raises privacy and regulatory concerns. In this work, we propose an alternative paradigm that bypasses target-domain training and recombines knowledge from existing datasets and models. Leveraging model merging and task arithmetic, we extract and combine task vectors from models fine-tuned on diverse public OVAR datasets. We show that, in out-of-distribution settings, the resulting merged model achieves superior zero-shot generalization to the pre-trained base model. Code is available at https://github.com/omaymaMoussadek/robust-ovar

Related papers

• Enhancing Linguistic Generalization of VLA: Fine-Tuning OpenVLA via Synthetic Instruction Augmentation [0.0]
  Generalization remains a core challenge in embodied AI, as robots must adapt to diverse environments.<n>This paper proposes a parameter-efficient fine-tuning strategy to enhance the linguistic generalization of OpenVLA.
  arXiv  Detail & Related papers  (2026-03-17T01:04:15Z)
• Toward Enhancing Representation Learning in Federated Multi-Task Settings [13.483094713610322]
  Federated multi-task learning (FMTL) seeks to collaboratively train customized models for users with different tasks.<n>We propose Muscle loss, a contrastive learning objective that simultaneously aligns representations from all participating models.<n>We develop FedMuscle, a practical and communication-efficient FMTL algorithm that naturally handles both model and task.
  arXiv  Detail & Related papers  (2026-02-02T04:39:36Z)
• Implicit Federated In-context Learning For Task-Specific LLM Fine-Tuning [10.042856500868805]
  We propose the Implicit Federated In-Context Learning (IFed-ICL) framework.<n>IFed-ICL draws inspiration from federated learning to establish a novel distributed collaborative paradigm.<n>Compared to traditional methods, IFed-ICL avoids the extensive parameter updates required by conventional fine-tuning methods.
  arXiv  Detail & Related papers  (2025-11-10T06:34:29Z)
• RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging [33.22889542330089]
  Internal representations in large language models (LLMs) serve as reliable proxies of learned knowledge.<n>We propose RECALL, a representation-aware model merging framework for continual learning without access to historical data.
  arXiv  Detail & Related papers  (2025-10-23T12:17:37Z)
• Enhanced Continual Learning of Vision-Language Models with Model Fusion [16.764069327701186]
  Vision-Language Models (VLMs) represent a breakthrough in artificial intelligence.<n>VLMs are susceptible to catastrophic forgetting when sequentially fine-tuned on multiple downstream tasks.<n>We propose Continual Decoupling-Unifying (ConDU), a novel approach, by introducing model fusion into continual learning.
  arXiv  Detail & Related papers  (2025-03-12T15:48:13Z)
• Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models [83.02797560769285]
  Data-Free Meta-Learning (DFML) aims to derive knowledge from a collection of pre-trained models without accessing their original data.<n>Current methods often overlook the heterogeneity among pre-trained models, which leads to performance degradation due to task conflicts.
  arXiv  Detail & Related papers  (2024-05-26T13:11:55Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models [55.5610165938949]
  Fine-tuning vision-language models (VLMs) has gained increasing popularity due to its practical value. This paper explores the collaborative potential of leveraging much weaker VLMs to enhance the generalization of a robust single model. We introduce three customized ensemble strategies, each tailored to one specific scenario. The proposed ensemble strategies are evaluated on zero-shot, base-to-new, and cross-dataset generalization, achieving new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-11-28T05:17:25Z)
• Toward Certified Robustness Against Real-World Distribution Shifts [65.66374339500025]
  We train a generative model to learn perturbations from data and define specifications with respect to the output of the learned model. A unique challenge arising from this setting is that existing verifiers cannot tightly approximate sigmoid activations. We propose a general meta-algorithm for handling sigmoid activations which leverages classical notions of counter-example-guided abstraction refinement.
  arXiv  Detail & Related papers  (2022-06-08T04:09:13Z)
• Towards Inheritable Models for Open-Set Domain Adaptation [56.930641754944915]
  We introduce a practical Domain Adaptation paradigm where a source-trained model is used to facilitate adaptation in the absence of the source dataset in future. We present an objective way to quantify inheritability to enable the selection of the most suitable source model for a given target domain, even in the absence of the source data.
  arXiv  Detail & Related papers  (2020-04-09T07:16:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.