TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition

• URL: http://arxiv.org/abs/2606.25478v2
• Date: Mon, 29 Jun 2026 04:17:52 GMT
• Title: TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognition
• Abstract summary: We argue that existing studies overlook the deviation of representations beyond the fine-tuning data distribution.<n>We introduce emphTACO, a framework to mitigate the potential negative effects induced by this inconsistency.
• Score: 35.164192530400356
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adapting CLIP for open-vocabulary video recognition necessitates a delicate balance between newly acquired video knowledge and the pretrained generalization. While existing studies pursue this generalization-specialization trade-off with additional regularizations or constraints, we argue that they overlook the deviation of representations beyond the fine-tuning data distribution, resulting in suboptimal adaptation effects. We believe such deviation is inherited from the inconsistency between the fine-tuning and evaluation objectives, where model optimization is restricted to the known training distribution but evaluated on unseen ones. In this paper, we introduce \emph{TACO}, a simple yet effective framework to mitigate the potential negative effects induced by this inconsistency. Our key insight is that adaptation should preserve OOD-relevant alignment beyond the training distribution. To this end, we propose \emph{Relative Structure Distillation}, which regularizes the relative geometry of the representation space and suppresses harmful alignment shift during training. We further decouple the representation space from the optimization space with a lightweight specialization projection, allowing task-specific adaptation without directly overspecializing the representations used at test time. \emph{TACO} establishes state-of-the-art performance on diverse benchmarks under cross-dataset and base-to-novel settings. Code will be released at https://github.com/ZMHH-H/TACO.

Related papers

• Unified Video Dense Prediction from Disjoint Data [71.9957801898161]
  We introduce UniD, a unified video model that jointly predicts eight dense scene properties-depth, surface normals, semantic segmentation, boundaries, human parts, albedo, shading, and materials.<n>We propose a simple yet effective distillation step in which per-task experts supervise a unified backbone through lightweight task projectors.<n>UniD achieves competitive performance against per-task specialists and multi-task baselines.
  arXiv  Detail & Related papers  (2026-07-23T17:59:50Z)
• GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution [5.770893169582546]
  This work reframes attribution as subset-level counterfactual utility prediction.<n>We introduce GRASP, an interaction-aware surrogate.<n>It decisively outperforms existing scalable baselines.
  arXiv  Detail & Related papers  (2026-06-05T04:17:50Z)
• Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning [25.20062959668559]
  This paper focuses on the target-domain few-shot finetuning in the CLIP-based CDFSL task.<n>We find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens improves target-domain performance.<n>Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP.
  arXiv  Detail & Related papers  (2026-05-28T11:21:44Z)
• One Last Attention for Your Vision-Language Model [42.872184600248914]
  We propose textbfRational textbfAdaptaion (RAda) to explicitly exploit the final fused representation during fine-tuning.<n> RAda employs a learned mask, obtained from a lightweight attention layer attached at the end of a VLM, to dynamically calibrate the contribution of each element in the rational matrix.<n>Experiments show that RAda serves as a versatile fine-tuning technique, improving the baseline with minimal code and performing comparably against current arts in most settings.
  arXiv  Detail & Related papers  (2025-07-21T10:35:32Z)
• Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation [5.296260279593993]
  Vision-language models (VLMs) such as CLIP demonstrate strong performance but struggle when adapted to downstream tasks.<n>We propose an optimal transport (OT)-guided prompt learning framework that mitigates forgetting by preserving the structural consistency of feature distributions.<n>Our approach enforces joint constraints on both vision and text representations, ensuring a holistic feature alignment.
  arXiv  Detail & Related papers  (2025-03-11T21:38:34Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Transform-Equivariant Consistency Learning for Temporal Sentence Grounding [66.10949751429781]
  We introduce a novel Equivariant Consistency Regulation Learning framework to learn more discriminative representations for each video. Our motivation comes from that the temporal boundary of the query-guided activity should be consistently predicted. In particular, we devise a self-supervised consistency loss module to enhance the completeness and smoothness of the augmented video.
  arXiv  Detail & Related papers  (2023-05-06T19:29:28Z)
• Proposal Distribution Calibration for Few-Shot Object Detection [65.19808035019031]
  In few-shot object detection (FSOD), the two-step training paradigm is widely adopted to mitigate the severe sample imbalance. Unfortunately, the extreme data scarcity aggravates the proposal distribution bias, hindering the RoI head from evolving toward novel classes. We introduce a simple yet effective proposal distribution calibration (PDC) approach to neatly enhance the localization and classification abilities of the RoI head.
  arXiv  Detail & Related papers  (2022-12-15T05:09:11Z)
• Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE [51.09507030387935]
  Wasserstein autoencoder (WAE) shows that matching two distributions is equivalent to minimizing a simple autoencoder (AE) loss under the constraint that the latent space of this AE matches a pre-specified prior distribution. We propose to use the contrastive learning framework that has been shown to be effective for self-supervised representation learning, as a means to resolve this problem. We show that using the contrastive learning framework to optimize the WAE loss achieves faster convergence and more stable optimization compared with existing popular algorithms for WAE.
  arXiv  Detail & Related papers  (2021-10-19T22:55:47Z)
• Efficient Semantic Image Synthesis via Class-Adaptive Normalization [116.63715955932174]
  Class-adaptive normalization (CLADE) is a lightweight but equally-effective variant that is only adaptive to semantic class. We introduce intra-class positional map encoding calculated from semantic layouts to modulate the normalization parameters of CLADE. The proposed CLADE can be generalized to different SPADE-based methods while achieving comparable generation quality compared to SPADE.
  arXiv  Detail & Related papers  (2020-12-08T18:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.