ContextRefine-CLIP for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2025

• URL: http://arxiv.org/abs/2506.10550v1
• Date: Thu, 12 Jun 2025 10:17:30 GMT
• Title: ContextRefine-CLIP for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2025
• Authors: Jing He, Yiqing Wang, Lingling Li, Kexin Zhang, Puhua Chen,
• Abstract summary: This report presents ContextRefine-CLIP, an efficient model for visual-textual multi-instance retrieval tasks.<n>The approach is based on the dual-encoder AVION, on which we introduce a cross-modal attention flow module.<n>The code will be released open-source on https://github.com/delCayr/ContextRefine-Clip.
• Score: 6.945344449218478
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This report presents ContextRefine-CLIP (CR-CLIP), an efficient model for visual-textual multi-instance retrieval tasks. The approach is based on the dual-encoder AVION, on which we introduce a cross-modal attention flow module to achieve bidirectional dynamic interaction and refinement between visual and textual features to generate more context-aware joint representations. For soft-label relevance matrices provided in tasks such as EPIC-KITCHENS-100, CR-CLIP can work with Symmetric Multi-Similarity Loss to achieve more accurate semantic alignment and optimization using the refined features. Without using ensemble learning, the CR-CLIP model achieves 66.78mAP and 82.08nDCG on the EPIC-KITCHENS-100 public leaderboard, which significantly outperforms the baseline model and fully validates its effectiveness in cross-modal retrieval. The code will be released open-source on https://github.com/delCayr/ContextRefine-Clip

Related papers

• Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation [4.063715077687089]
  Distill CLIP (DCLIP) is a fine-tuned variant of the CLIP model.<n>It enhances multimodal image-text retrieval while preserving the original model's strong zero-shot classification capabilities.
  arXiv  Detail & Related papers  (2025-05-25T07:08:07Z)
• PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition [7.966499123076283]
  Vision-Language Models (VLMs) like CLIP offer promising solutions for Dynamic Facial Expression Recognition (DFER)<n>We propose PE-CLIP, a parameter-efficient fine-tuning framework that adapts CLIP for DFER while significantly reducing trainable parameters.<n>By balancing efficiency and accuracy, PE-CLIP sets a new benchmark in resource-efficient DFER.
  arXiv  Detail & Related papers  (2025-03-21T08:45:50Z)
• DiffCLIP: Differential Attention Meets CLIP [57.396578974401734]
  We propose DiffCLIP, a novel vision-language model that extends the differential attention mechanism to CLIP architectures.<n>With minimal additional parameters, DiffCLIP achieves superior performance on image-text understanding tasks.
  arXiv  Detail & Related papers  (2025-03-09T14:04:09Z)
• FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs [5.35588281968644]
  We propose a novel framework, named Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERs (Fine CLIPER)<n>Our Fine CLIPER achieves tunable SOTA performance on the DFEW, FERV39k, and MAFW datasets with few parameters.
  arXiv  Detail & Related papers  (2024-07-02T10:55:43Z)
• CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling [53.97609687516371]
  Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data.<n>Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates.<n>We propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling.
  arXiv  Detail & Related papers  (2024-06-25T12:47:04Z)
• Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification [54.96876797812238]
  We present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts. The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images. Experiment results across various datasets and models confirm CODER's effectiveness.
  arXiv  Detail & Related papers  (2024-04-27T02:04:36Z)
• Towards More Unified In-context Visual Understanding [74.55332581979292]
  We present a new ICL framework for visual understanding with multi-modal output enabled. First, we quantize and embed both text and visual prompt into a unified representational space. Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them.
  arXiv  Detail & Related papers  (2023-12-05T06:02:21Z)
• Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features [32.138956674478116]
  Given a query composed of a reference image and a relative caption, the Composed Image Retrieval goal is to retrieve images visually similar to the reference one. We use features from the OpenAI CLIP model to tackle the considered task. We train a Combiner network that learns to combine the image-text features integrating the bimodal information.
  arXiv  Detail & Related papers  (2023-08-22T15:03:16Z)
• MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations [14.355743915598554]
  We propose MXM-CLR, a unified framework for contrastive learning of multifold cross-modal representations. XM-CLR explicitly models and learns the relationships between multifold observations of instances from different modalities. Results show the superiority of MXM-CLR in learning better representations for the multifold data.
  arXiv  Detail & Related papers  (2023-03-20T02:51:53Z)
• Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training [88.80694147730883]
  We investigate a variety of Modality-Shared Contrastive Language-Image Pre-training (MS-CLIP) frameworks. In studied conditions, we observe that a mostly unified encoder for vision and language signals outperforms all other variations that separate more parameters. Our approach outperforms vanilla CLIP by 1.6 points in linear probing on a collection of 24 downstream vision tasks.
  arXiv  Detail & Related papers  (2022-07-26T05:19:16Z)
• Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval [152.3504607706575]
  This research aims to conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories. We first contribute the Product1M datasets, and define two real practical instance-level retrieval tasks. We exploit to train a more effective cross-modal model which is adaptively capable of incorporating key concept information from the multi-modal data.
  arXiv  Detail & Related papers  (2022-06-17T15:40:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.