Visual Navigation of Digital Libraries: Retrieval and Classification of Images in the National Library of Norway's Digitised Book Collection

• URL: http://arxiv.org/abs/2410.14969v1
• Date: Sat, 19 Oct 2024 04:20:23 GMT
• Title: Visual Navigation of Digital Libraries: Retrieval and Classification of Images in the National Library of Norway's Digitised Book Collection
• Authors: Marie Roald, Magnus Breder Birkenes, Lars Gunnarsønn Bagøien Johnsen,
• Abstract summary: We present a proof-of-concept image search application for exploring images in the National Library of Norway's pre-1900 books. We compare Vision Transformer (ViT), Contrastive Language-Image Pre-training (CLIP), and Sigmoid loss for Language-Image Pre-training (SigLIP) embeddings for image retrieval and classification.
• Score: 0.3277163122167433
• License:
• Abstract: Digital tools for text analysis have long been essential for the searchability and accessibility of digitised library collections. Recent computer vision advances have introduced similar capabilities for visual materials, with deep learning-based embeddings showing promise for analysing visual heritage. Given that many books feature visuals in addition to text, taking advantage of these breakthroughs is critical to making library collections open and accessible. In this work, we present a proof-of-concept image search application for exploring images in the National Library of Norway's pre-1900 books, comparing Vision Transformer (ViT), Contrastive Language-Image Pre-training (CLIP), and Sigmoid loss for Language-Image Pre-training (SigLIP) embeddings for image retrieval and classification. Our results show that the application performs well for exact image retrieval, with SigLIP embeddings slightly outperforming CLIP and ViT in both retrieval and classification tasks. Additionally, SigLIP-based image classification can aid in cleaning image datasets from a digitisation pipeline.

Related papers

• Transductive Learning for Near-Duplicate Image Detection in Scanned Photo Collections [0.0]
  This paper presents a comparative study of near-duplicate image detection techniques in a real-world use case scenario. We propose a transductive learning approach that leverages state-of-the-art deep learning architectures such as convolutional neural networks (CNNs) and Vision Transformers (ViTs) The results show that the proposed approach outperforms the baseline methods in the task of near-duplicate image detection in the UKBench and an in-house private dataset.
  arXiv  Detail & Related papers  (2024-10-25T09:56:15Z)
• Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models [2.3301643766310374]
  By utilizing multi-modal large language models (M-LLMs) that support visual prompting, we can extract image features and convert them into textual data. We show the superior precision and recall performance of our image retrieval method compared to conventional vision-language model-based methods. We also demonstrate that the retrieval performance can be improved by iteratively incorporating keywords into search queries.
  arXiv  Detail & Related papers  (2024-08-29T06:54:03Z)
• Learning text-to-video retrieval from image captioning [59.81537951811595]
  We describe a protocol to study text-to-video retrieval training with unlabeled videos. We assume (i) no access to labels for any videos, and (ii) access to labeled images in the form of text. We show that automatically labeling video frames with image captioning allows text-to-video retrieval training.
  arXiv  Detail & Related papers  (2024-04-26T15:56:08Z)
• Enhancing Image Retrieval : A Comprehensive Study on Photo Search using the CLIP Mode [0.27195102129095]
  Photo search has witnessed significant advancements with the introduction of CLIP (Contrastive Language-Image Pretraining) model. This abstract summarizes the foundational principles of CLIP and highlights its potential impact on advancing the field of photo search.
  arXiv  Detail & Related papers  (2024-01-24T17:35:38Z)
• SILC: Improving Vision Language Pretraining with Self-Distillation [113.50400246862056]
  We introduce SILC, a novel framework for vision language pretraining. SILC improves image-text contrastive learning with the simple addition of local-to-global correspondence learning by self-distillation. We show that distilling local image features from an exponential moving average (EMA) teacher model significantly improves model performance on dense predictions tasks like detection and segmentation.
  arXiv  Detail & Related papers  (2023-10-20T08:44:47Z)
• Constructing Image-Text Pair Dataset from Books [10.92677060085447]
  We propose a novel approach to leverage digital archives for machine learning. In our experiments, we apply our pipeline on old photo books to construct an image-text pair dataset.
  arXiv  Detail & Related papers  (2023-10-03T10:23:28Z)
• Non-Contrastive Learning Meets Language-Image Pre-Training [145.6671909437841]
  We study the validity of non-contrastive language-image pre-training (nCLIP) We introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics.
  arXiv  Detail & Related papers  (2022-10-17T17:57:46Z)
• LAVIS: A Library for Language-Vision Intelligence [98.88477610704938]
  LAVIS is an open-source library for LAnguage-VISion research and applications. It features a unified interface to easily access state-of-the-art image-language, video-language models and common datasets.
  arXiv  Detail & Related papers  (2022-09-15T18:04:10Z)
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision [57.031588264841]
  We leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme.
  arXiv  Detail & Related papers  (2021-02-11T10:08:12Z)
• PyRetri: A PyTorch-based Library for Unsupervised Image Retrieval by Deep Convolutional Neural Networks [49.35908338404728]
  PyRetri is an open source library for deep learning based unsupervised image retrieval. It encapsulates the retrieval process in several stages and provides functionality that covers various prominent methods for each stage.
  arXiv  Detail & Related papers  (2020-05-02T10:17:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.