Is Medieval Distant Viewing Possible? : Extending and Enriching Annotation of Legacy Image Collections using Visual Analytics

• URL: http://arxiv.org/abs/2208.09657v2
• Date: Thu, 11 Apr 2024 12:25:45 GMT
• Title: Is Medieval Distant Viewing Possible? : Extending and Enriching Annotation of Legacy Image Collections using Visual Analytics
• Authors: Christofer Meinecke, Estelle Guéville, David Joseph Wrisley, Stefan Jänicke,
• Abstract summary: We describe working with two pre-annotated sets of medieval manuscript images that exhibit conflicting and overlapping metadata. We aim to create a more uniform set of labels to serve as a "bridge" in the combined dataset. Visual interfaces provide experts an overview of relationships in the data going beyond the sum total of the metadata.
• Score: 3.89394670917253
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Distant viewing approaches have typically used image datasets close to the contemporary image data used to train machine learning models. To work with images from other historical periods requires expert annotated data, and the quality of labels is crucial for the quality of results. Especially when working with cultural heritage collections that contain myriad uncertainties, annotating data, or re-annotating, legacy data is an arduous task. In this paper, we describe working with two pre-annotated sets of medieval manuscript images that exhibit conflicting and overlapping metadata. Since a manual reconciliation of the two legacy ontologies would be very expensive, we aim (1) to create a more uniform set of descriptive labels to serve as a "bridge" in the combined dataset, and (2) to establish a high quality hierarchical classification that can be used as a valuable input for subsequent supervised machine learning. To achieve these goals, we developed visualization and interaction mechanisms, enabling medievalists to combine, regularize and extend the vocabulary used to describe these, and other cognate, image datasets. The visual interfaces provide experts an overview of relationships in the data going beyond the sum total of the metadata. Word and image embeddings as well as co-occurrences of labels across the datasets, enable batch re-annotation of images, recommendation of label candidates and support composing a hierarchical classification of labels.

Related papers

• Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data [95.0476489266988]
  We present a novel data-efficient semi-supervised framework to improve the generalization of image captioning models. Our proposed method trains a captioner to learn from a paired data and to progressively associate unpaired data. Our extensive and comprehensive empirical results both on (1) image-based and (2) dense region-based captioning datasets followed by comprehensive analysis on the scarcely-paired dataset.
  arXiv  Detail & Related papers  (2023-01-26T15:25:43Z)
• Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation [2.2807344448218503]
  We propose a data-centric image classification benchmark with nine real-world datasets and multiple annotations per image. We show that multiple annotations allow a better approximation of the real underlying class distribution. We identify that hard labels can not capture the ambiguity of the data and this might lead to the common issue of overconfident models.
  arXiv  Detail & Related papers  (2022-07-13T14:17:21Z)
• Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph [96.95815946327079]
  It is difficult to learn the association between named entities and visual cues due to the long-tail distribution of named entities. We propose a novel approach that constructs a multi-modal knowledge graph to associate the visual objects with named entities.
  arXiv  Detail & Related papers  (2021-07-26T05:50:41Z)
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization [112.68171734288237]
  We propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. We learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization.
  arXiv  Detail & Related papers  (2021-04-12T21:41:25Z)
• Grafit: Learning fine-grained image representations with coarse labels [114.17782143848315]
  This paper tackles the problem of learning a finer representation than the one provided by training labels. By jointly leveraging the coarse labels and the underlying fine-grained latent space, it significantly improves the accuracy of category-level retrieval methods.
  arXiv  Detail & Related papers  (2020-11-25T19:06:26Z)
• Multi-label Zero-shot Classification by Learning to Transfer from External Knowledge [36.04579549557464]
  Multi-label zero-shot classification aims to predict multiple unseen class labels for an input image. This paper introduces a novel multi-label zero-shot classification framework by learning to transfer from external knowledge.
  arXiv  Detail & Related papers  (2020-07-30T17:26:46Z)
• Hierarchical Image Classification using Entailment Cone Embeddings [68.82490011036263]
  We first inject label-hierarchy knowledge into an arbitrary CNN-based classifier. We empirically show that availability of such external semantic information in conjunction with the visual semantics from images boosts overall performance.
  arXiv  Detail & Related papers  (2020-04-02T10:22:02Z)
• Learning Representations For Images With Hierarchical Labels [1.3579420996461438]
  We present a set of methods to leverage information about the semantic hierarchy induced by class labels. We show that availability of such external semantic information in conjunction with the visual semantics from images boosts overall performance. Although, both the CNN-classifiers injected with hierarchical information, and the embedding-based models outperform a hierarchy-agnostic model on the newly presented, real-world ETH Entomological Collection image dataset.
  arXiv  Detail & Related papers  (2020-04-02T09:56:03Z)
• Collaborative Learning of Semi-Supervised Clustering and Classification for Labeling Uncurated Data [6.871887763122593]
  Domain-specific image collections present potential value in various areas of science and business. To employ contemporary supervised image analysis methods on such image data, they must first be cleaned and organized, and then manually labeled for the nomenclature employed in the specific domain. We designed and implemented the Plud system to minimize the effort spent by an expert and handles realistic large collections of images.
  arXiv  Detail & Related papers  (2020-03-09T17:03:05Z)
• Automatically Discovering and Learning New Visual Categories with Ranking Statistics [145.89790963544314]
  We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. We learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data. We evaluate our approach on standard classification benchmarks and outperform current methods for novel category discovery by a significant margin.
  arXiv  Detail & Related papers  (2020-02-13T18:53:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.