Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-Relational Representations

• URL: http://arxiv.org/abs/2607.16321v1
• Date: Wed, 15 Jul 2026 09:28:48 GMT
• Title: Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-Relational Representations
• Abstract summary: We introduce CANVAS, a framework for learning relation-aware multimodal representations inspired by sheaf theory.<n>We evaluate on three newly introduced benchmarks of artworks for multi-relational art understanding.
• Score: 21.379603872788127
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding a painting is never a single act. Art historians may analyze the same work through concepts of style, iconography, or historical context, dimensions that are not interchangeable, and each carries distinct semantic relationships between the visual and the textual. Vision-Language Models (VLMs) like CLIP, which learn a single shared embedding space, collapse this richness into a single homogeneous alignment, thereby losing the multi-relational structure that defines art-historical reasoning. We introduce CANVAS (Contrastive Art-aware Network for Vision-Language Alignment with Sheaves), a framework for learning relation-aware multimodal representations inspired by sheaf theory. Each artwork is projected into multiple embeddings conditioned on the type of relation (i.e., the context), and a novel contrastive loss encodes contextual information during training, with no dependency on external data at inference. We evaluate on three newly introduced benchmarks of artworks for multi-relational art understanding: WikiArt+, derived from WikiArt and Wikipedia, HertzianaDP, from the Bibliotheca Hertziana collection, and SemArt+, refined from the SemArt dataset. In multimodal retrieval and art understanding, CANVAS outperforms the baselines, supporting the view that multi-relational alignment is not just theoretically motivated but also practically essential.

Related papers

• Conducting Stylistic Analysis of Paintings through an Art-History Agent [0.0]
  We present an AI framework that automates stylistic analysis of paintings.<n>By training a vision transformer (ViT) on a large corpus of paintings with metadata, our system encodes this art history-specific data as embeddings.<n>A large language model (LLM) then interprets each feature by retrieving associated artworks and their accompanying curator-written texts, and synthesizes them into descriptions.
  arXiv  Detail & Related papers  (2026-08-30T08:10:58Z)
• Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style [60.562013130566726]
  We characterize the mechanisms underlying VLMs' ability to predict artistic style.<n>We employ a latent-space decomposition approach to identify concepts that drive art style prediction.<n>In cases where an irrelevant concept was used to successfully predict style, art historians identified possible reasons for its success.
  arXiv  Detail & Related papers  (2026-03-11T17:49:45Z)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees [59.16312652369709]
  Contrastive Language-Image Pre-training (CLIP)citepradford2021learning has emerged as a pivotal model in computer vision and multimodal learning.<n>CLIP struggles with potential information misalignment in many image-text datasets and suffers from entangled representation.<n>We introduce ours, a novel approach that identifies and aligns the most relevant visual and textual representations in a modular manner.
  arXiv  Detail & Related papers  (2025-07-29T22:26:20Z)
• ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval [8.94249680213101]
  ArtSeek is a framework for art analysis that combines multimodal large language models with retrieval-augmented generation.<n>ArtSeek integrates three key components: an intelligent multimodal retrieval module based on late interaction retrieval, a contrastive multitask classification network for predicting artist, genre, style, media, and tags, and an agentic reasoning strategy.<n>Our framework achieves state-of-the-art results on multiple benchmarks, including a +8.4% F1 improvement in style classification over GraphCLIP and a +7.1 BLEU@1 gain in captioning on ArtPedia.
  arXiv  Detail & Related papers  (2025-07-29T15:31:58Z)
• ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding [21.469839845171652]
  ArtRAG is a novel framework that combines structured knowledge with retrieval-augmented generation (RAG) for multi-perspective artwork explanation.<n>At inference time, a structured retriever selects semantically and topologically relevant subgraphs to guide generation.<n>Experiments on the SemArt and Artpedia datasets show that ArtRAG outperforms several heavily trained baselines.
  arXiv  Detail & Related papers  (2025-05-09T13:08:27Z)
• Does CLIP perceive art the same way we do? [0.05833117322405446]
  We investigate CLIP's ability to extract high-level semantic and stylistic information from paintings.<n>Our findings reveal both strengths and limitations in CLIP's visual representations.<n>Our work highlights the need for deeper interpretability in multimodal systems.
  arXiv  Detail & Related papers  (2025-05-08T13:21:10Z)
• Compositional Entailment Learning for Hyperbolic Vision-Language Models [54.41927525264365]
  We show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs.<n>We propose Compositional Entailment Learning for hyperbolic vision-language models.<n> Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning.
  arXiv  Detail & Related papers  (2024-10-09T14:12:50Z)
• KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph [24.586916324061168]
  We present KALE Knowledge-Augmented vision-Language model for artwork Elaborations. KALE incorporates the metadata in two ways: firstly as direct textual input, and secondly through a multimodal heterogeneous knowledge graph. Experimental results demonstrate that KALE achieves strong performance over existing state-of-the-art work across several artwork datasets.
  arXiv  Detail & Related papers  (2024-09-17T06:39:18Z)
• CreativeSynth: Cross-Art-Attention for Artistic Image Synthesis with Multimodal Diffusion [73.08710648258985]
  Key painting attributes including layout, perspective, shape, and semantics often cannot be conveyed and expressed through style transfer.<n>Large-scale pretrained text-to-image generation models have demonstrated their capability to synthesize a vast amount of high-quality images.<n>Our main novel idea is to integrate multimodal semantic information as a synthesis guide into artworks, rather than transferring style to the real world.
  arXiv  Detail & Related papers  (2024-01-25T10:42:09Z)
• Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality [50.48859793121308]
  Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning. Recent research has highlighted severe limitations in their ability to perform compositional reasoning over objects, attributes, and relations.
  arXiv  Detail & Related papers  (2023-05-23T08:28:38Z)
• ALADIN-NST: Self-supervised disentangled representation learning of artistic style through Neural Style Transfer [60.6863849241972]
  We learn a representation of visual artistic style more strongly disentangled from the semantic content depicted in an image. We show that strongly addressing the disentanglement of style and content leads to large gains in style-specific metrics.
  arXiv  Detail & Related papers  (2023-04-12T10:33:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.