Heri-Graphs: A Workflow of Creating Datasets for Multi-modal Machine Learning on Graphs of Heritage Values and Attributes with Social Media

• URL: http://arxiv.org/abs/2205.07545v1
• Date: Mon, 16 May 2022 09:45:45 GMT
• Title: Heri-Graphs: A Workflow of Creating Datasets for Multi-modal Machine Learning on Graphs of Heritage Values and Attributes with Social Media
• Authors: Nan Bai, Pirouz Nourian, Renqian Luo, Ana Pereira Roders
• Abstract summary: Values (why to conserve) and Attributes (what to conserve) are essential concepts of cultural heritage. Recent studies have been using social media to map values and attributes conveyed by public to cultural heritage. This study presents a methodological workflow for constructing such multi-modal datasets using posts and images on Flickr.
• Score: 7.318997639507268
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Values (why to conserve) and Attributes (what to conserve) are essential concepts of cultural heritage. Recent studies have been using social media to map values and attributes conveyed by public to cultural heritage. However, it is rare to connect heterogeneous modalities of images, texts, geo-locations, timestamps, and social network structures to mine the semantic and structural characteristics therein. This study presents a methodological workflow for constructing such multi-modal datasets using posts and images on Flickr for graph-based machine learning (ML) tasks concerning heritage values and attributes. After data pre-processing using state-of-the-art ML models, the multi-modal information of visual contents and textual semantics are modelled as node features and labels, while their social relationships and spatiotemporal contexts are modelled as links in Multi-Graphs. The workflow is tested in three cities containing UNESCO World Heritage properties - Amsterdam, Suzhou, and Venice, which yielded datasets with high consistency for semi-supervised learning tasks. The entire process is formally described with mathematical notations, ready to be applied in provisional tasks both as ML problems with technical relevance and as urban/heritage study questions with societal interests. This study could also benefit the understanding and mapping of heritage values and attributes for future research in global cases, aiming at inclusive heritage management practices.

Related papers

• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• ARMADA: Attribute-Based Multimodal Data Augmentation [93.05614922383822]
  Attribute-based Multimodal Data Augmentation (ARMADA) is a novel multimodal data augmentation method via knowledge-guided manipulation of visual attributes. ARMADA is a novel multimodal data generation framework that: (i) extracts knowledge-grounded attributes from symbolic KBs for semantically consistent yet distinctive image-text pair generation. This also highlights the need to leverage external knowledge proxies for enhanced interpretability and real-world grounding.
  arXiv  Detail & Related papers  (2024-08-19T15:27:25Z)
• MaterioMiner -- An ontology-based text mining dataset for extraction of process-structure-property entities [0.0]
  We present the MaterioMiner dataset and the materials ontology where ontological concepts are associated with textual entities. We explore the consistency between the three raters and perform fine-process-trained models to showcase the feasibility of named-process recognition model training.
  arXiv  Detail & Related papers  (2024-08-05T21:42:59Z)
• Bridging Local Details and Global Context in Text-Attributed Graphs [62.522550655068336]
  GraphBridge is a framework that bridges local and global perspectives by leveraging contextual textual information. Our method achieves state-of-theart performance, while our graph-aware token reduction module significantly enhances efficiency and solves scalability issues.
  arXiv  Detail & Related papers  (2024-06-18T13:35:25Z)
• Multimodal Metadata Assignment for Cultural Heritage Artifacts [1.5826261914050386]
  We develop a multimodal classifier for the cultural heritage domain using a late fusion approach. The three modalities are Image, Text, and Tabular data. All individual classifiers accurately predict missing properties in the digitized silk artifacts, with the multimodal approach providing the best results.
  arXiv  Detail & Related papers  (2024-06-01T12:41:03Z)
• From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models [98.41645229835493]
  Data visualization in the form of charts plays a pivotal role in data analysis, offering critical insights and aiding in informed decision-making. Large foundation models, such as large language models, have revolutionized various natural language processing tasks. This survey paper serves as a comprehensive resource for researchers and practitioners in the fields of natural language processing, computer vision, and data analysis.
  arXiv  Detail & Related papers  (2024-03-18T17:57:09Z)
• Contextualized Structural Self-supervised Learning for Ontology Matching [0.9402105308876642]
  We introduce a novel self-supervised learning framework called LaKERMap. LaKERMap capitalizes on the contextual and structural information of concepts by integrating implicit knowledge into transformers. The findings from our innovative approach reveal that LaKERMap surpasses state-of-the-art systems in terms of alignment quality and inference time.
  arXiv  Detail & Related papers  (2023-10-05T18:51:33Z)
• Exploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Text [3.114960935006655]
  This paper aims to improve the state of the art of automatic construction and completion of knowledge graphs from text. In this context, one emerging paradigm is in-context learning where a language model is used as it is with a prompt.
  arXiv  Detail & Related papers  (2023-05-15T17:10:19Z)
• Can I see an Example? Active Learning the Long Tail of Attributes and Relations [64.50739983632006]
  We introduce a novel incremental active learning framework that asks for attributes and relations in visual scenes. While conventional active learning methods ask for labels of specific examples, we flip this framing to allow agents to ask for examples from specific categories. Using this framing, we introduce an active sampling method that asks for examples from the tail of the data distribution and show that it outperforms classical active learning methods on Visual Genome.
  arXiv  Detail & Related papers  (2022-03-11T19:28:19Z)
• GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph [32.129005474301735]
  Contextual information is often the key to structure such real world data, and we propose to use it in form of a knowledge graph. We propose a novel use of a knowledge graph, that is constructed on annotated data and pseudo-labeled data. With label propagation, we boost artwork classification by training a model using a graph convolutional network.
  arXiv  Detail & Related papers  (2021-05-25T11:50:05Z)
• KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion [99.47414073164656]
  A comprehensive knowledge graph (KG) contains an instance-level entity graph and an ontology-level concept graph. The two-view KG provides a testbed for models to "simulate" human's abilities on knowledge abstraction, concretization, and completion. We propose a unified KG benchmark by improving existing benchmarks in terms of dataset scale, task coverage, and difficulty.
  arXiv  Detail & Related papers  (2020-04-28T16:21:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.