WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

• URL: http://arxiv.org/abs/2603.09921v1
• Date: Tue, 10 Mar 2026 17:18:53 GMT
• Title: WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition
• Abstract summary: Open-domain visual entity recognition (VER) seeks to associate images with entities in encyclopedic knowledge bases such as Wikipedia.<n>Recent generative methods tailored for VER demonstrate strong performance but incur high computational costs.<n>We introduce WikiCLIP, a framework that establishes a strong and efficient baseline for open-domain VER.
• Score: 18.56932287056642
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-domain visual entity recognition (VER) seeks to associate images with entities in encyclopedic knowledge bases such as Wikipedia. Recent generative methods tailored for VER demonstrate strong performance but incur high computational costs, limiting their scalability and practical deployment. In this work, we revisit the contrastive paradigm for VER and introduce WikiCLIP, a simple yet effective framework that establishes a strong and efficient baseline for open-domain VER. WikiCLIP leverages large language model embeddings as knowledge-rich entity representations and enhances them with a Vision-Guided Knowledge Adaptor (VGKA) that aligns textual semantics with visual cues at the patch level. To further encourage fine-grained discrimination, a Hard Negative Synthesis Mechanism generates visually similar but semantically distinct negatives during training. Experimental results on popular open-domain VER benchmarks, such as OVEN, demonstrate that WikiCLIP significantly outperforms strong baselines. Specifically, WikiCLIP achieves a 16% improvement on the challenging OVEN unseen set, while reducing inference latency by nearly 100 times compared with the leading generative model, AutoVER. The project page is available at https://artanic30.github.io/project_pages/WikiCLIP/

Related papers

• SuperCLIP: CLIP with Simple Classification Supervision [88.86549733903314]
  Contrastive Language-Image Pretraining achieves strong generalization in vision-language tasks by aligning images and texts in a shared embedding space.<n>Recent findings show that CLIP-like models still underutilize fine-grained semantic signals in text.<n>We propose SuperCLIP, a framework that augments contrastive learning with classification-based supervision.
  arXiv  Detail & Related papers  (2025-12-16T15:11:53Z)
• Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning [17.580250180523752]
  Open-domain visual entity recognition aims to identify and link entities depicted in images to a vast and evolving set of real-world concepts.<n>We propose a Knowledge-guided Contrastive Learning framework that combines both images and text descriptions into a shared semantic space.<n>Our experiments show that using visual, textual, and structured knowledge greatly improves accuracy, especially for rare and unseen entities.
  arXiv  Detail & Related papers  (2025-10-15T15:33:36Z)
• Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits [59.66134971408414]
  We aim to adapt CLIP for fine-grained and structured understanding of museum exhibits. Our dataset is the first of its kind in the public domain. The proposed method (MUZE) learns to map CLIP's image embeddings to the tabular structure by means of a proposed transformer-based parsing network (parseNet)
  arXiv  Detail & Related papers  (2024-09-03T08:13:06Z)
• Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-hoc Retrieval [21.262531222066208]
  In this paper, we devise four pre-training objectives tailored for information retrieval tasks based on the structured knowledge of Wikipedia. Compared to existing pre-training methods, our approach can better capture the semantic knowledge in the training corpus. Experimental results in biomedical and legal domains demonstrate that our approach achieves better performance in vertical domains.
  arXiv  Detail & Related papers  (2023-12-17T09:31:47Z)
• VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection [58.47940430618352]
  We propose VadCLIP, a new paradigm for weakly supervised video anomaly detection (WSVAD) VadCLIP makes full use of fine-grained associations between vision and language on the strength of CLIP. We conduct extensive experiments on two commonly-used benchmarks, demonstrating that VadCLIP achieves the best performance on both coarse-grained and fine-grained WSVAD.
  arXiv  Detail & Related papers  (2023-08-22T14:58:36Z)
• Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities [54.26896306906937]
  We present OVEN-Wiki, where a model need to link an image onto a Wikipedia entity with respect to a text query. We show that a PaLI-based auto-regressive visual recognition model performs surprisingly well, even on Wikipedia entities that have never been seen during fine-tuning. While PaLI-based models obtain higher overall performance, CLIP-based models are better at recognizing tail entities.
  arXiv  Detail & Related papers  (2023-02-22T05:31:26Z)
• Unifying Graph Contrastive Learning with Flexible Contextual Scopes [57.86762576319638]
  We present a self-supervised learning method termed Unifying Graph Contrastive Learning with Flexible Contextual Scopes (UGCL for short) Our algorithm builds flexible contextual representations with contextual scopes by controlling the power of an adjacency matrix. Based on representations from both local and contextual scopes, distL optimises a very simple contrastive loss function for graph representation learning.
  arXiv  Detail & Related papers  (2022-10-17T07:16:17Z)
• Patch-level Representation Learning for Self-supervised Vision Transformers [68.8862419248863]
  Vision Transformers (ViTs) have gained much attention recently as a better architectural choice, often outperforming convolutional networks for various visual tasks. Inspired by this, we design a simple yet effective visual pretext task, coined SelfPatch, for learning better patch-level representations. We demonstrate that SelfPatch can significantly improve the performance of existing SSL methods for various visual tasks.
  arXiv  Detail & Related papers  (2022-06-16T08:01:19Z)
• Structural and Semantic Contrastive Learning for Self-supervised Node Representation Learning [32.126228702554144]
  Graph Contrastive Learning (GCL) has drawn much research interest for learning generalizable, transferable, and robust node representations in a self-supervised fashion. In this work, we go beyond the existing unsupervised GCL counterparts and address their limitations by proposing a simple yet effective framework S$3$-CL. Our experiments demonstrate that the node representations learned by S$3$-CL achieve superior performance on different downstream tasks compared to the state-of-the-art GCL methods.
  arXiv  Detail & Related papers  (2022-02-17T07:20:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.