Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models

• URL: http://arxiv.org/abs/2505.10583v1
• Date: Wed, 14 May 2025 09:41:38 GMT
• Title: Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models
• Authors: Diogo Freitas, Brigt Håvardstun, Cèsar Ferri, Darío Garigliotti, Jan Arne Telle, José Hernández-Orallo,
• Abstract summary: We evaluate the complexity of teaching visual-language models using two presentations: raw images as bitmaps and trace coordinates in TikZ format.<n>The results indicate that image-based representations generally require fewer segments and achieve higher accuracy than coordinate-based representations.<n>But, surprisingly, the teaching size usually ranks concepts similarly across both modalities, even when controlling for (a human proxy of) concept priors.
• Score: 12.127373862889176
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models have become multimodal, and many of them are said to integrate their modalities using common representations. If this were true, a drawing of a car as an image, for instance, should map to the similar area in the latent space as a textual description of the strokes that conform the drawing. To explore this in a black-box access regime to these models, we propose the use of machine teaching, a theory that studies the minimal set of examples a teacher needs to choose so that the learner captures the concept. In this paper we evaluate the complexity of teaching visual-language models a subset of objects in the Quick, Draw! dataset using two presentations: raw images as bitmaps and trace coordinates in TikZ format. The results indicate that image-based representations generally require fewer segments and achieve higher accuracy than coordinate-based representations. But, surprisingly, the teaching size usually ranks concepts similarly across both modalities, even when controlling for (a human proxy of) concept priors, suggesting that the simplicity of concepts may be an inherent property that transcends modality representations.

Related papers

• Multimodal Representation Alignment for Cross-modal Information Retrieval [12.42313654539524]
  Different machine learning models can represent the same underlying concept in different ways.<n>This variability is particularly valuable for in-the-wild multimodal retrieval, where the objective is to identify the corresponding representation in one modality given another as input.<n>In this work, we first investigate the geometric relationships between visual and textual embeddings derived from both vision-language models and combined unimodal models.<n>We then align these representations using four standard similarity metrics as well as two learned ones, implemented via neural networks.
  arXiv  Detail & Related papers  (2025-06-10T13:16:26Z)
• SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches [116.1810651297801]
  SketchYourSeg establishes freehand sketches as a powerful query modality for subjective image segmentation.<n>Our evaluations demonstrate superior performance over existing approaches across diverse benchmarks.
  arXiv  Detail & Related papers  (2025-01-27T13:07:51Z)
• Compositional Entailment Learning for Hyperbolic Vision-Language Models [54.41927525264365]
  We show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs.<n>We propose Compositional Entailment Learning for hyperbolic vision-language models.<n> Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning.
  arXiv  Detail & Related papers  (2024-10-09T14:12:50Z)
• Subobject-level Image Tokenization [60.80949852899857]
  Patch-based image tokenization ignores the morphology of the visual world.<n>Inspired by subword tokenization, we introduce subobject-level adaptive token segmentation.<n>We show that subobject tokenization enables faster convergence and better generalization while using fewer visual tokens.
  arXiv  Detail & Related papers  (2024-02-22T06:47:44Z)
• Keyword Spotting Simplified: A Segmentation-Free Approach using Character Counting and CTC re-scoring [8.6134769826665]
  Recent advances in segmentation-free keyword spotting treat this problem w.r.t. as an object detection paradigm. We propose a novel segmentation-free system that efficiently scans a document image to find rectangular areas that include the query information.
  arXiv  Detail & Related papers  (2023-08-07T12:11:04Z)
• Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation [38.09501948846373]
  We introduce the topological adjacency relationship among the tokens, emphasizing their relative position information. We incorporate these graphs into the pre-trained language model by adding graph neural network layers on top of the language model embeddings. Experiments on two benchmark datasets show that LAGER significantly outperforms strong baselines under different few-shot settings.
  arXiv  Detail & Related papers  (2023-05-24T07:34:33Z)
• Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens [76.40196364163663]
  We propose a learning-based vision-language pre-training approach, such as CLIP. We show that our method can learn more comprehensive representations and capture meaningful cross-modal correspondence.
  arXiv  Detail & Related papers  (2023-03-27T00:58:39Z)
• Correlational Image Modeling for Self-Supervised Visual Pre-Training [81.82907503764775]
  Correlational Image Modeling is a novel and surprisingly effective approach to self-supervised visual pre-training. Three key designs enable correlational image modeling as a nontrivial and meaningful self-supervisory task.
  arXiv  Detail & Related papers  (2023-03-22T15:48:23Z)
• Dense Semantic Contrast for Self-Supervised Visual Representation Learning [12.636783522731392]
  We present Dense Semantic Contrast (DSC) for modeling semantic category decision boundaries at a dense level. We propose a dense cross-image semantic contrastive learning framework for multi-granularity representation learning. Experimental results show that our DSC model outperforms state-of-the-art methods when transferring to downstream dense prediction tasks.
  arXiv  Detail & Related papers  (2021-09-16T07:04:05Z)
• Accurate Word Representations with Universal Visual Guidance [55.71425503859685]
  This paper proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. We build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach.
  arXiv  Detail & Related papers  (2020-12-30T09:11:50Z)
• Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning [116.91819311885166]
  We propose a hierarchical semantic alignment strategy via expanding the views generated by a single image to textbfCross-samples and Multi-level representation. Our method, termed as CsMl, has the ability to integrate multi-level visual representations across samples in a robust way.
  arXiv  Detail & Related papers  (2020-12-04T17:26:24Z)
• Learning to Represent Image and Text with Denotation Graph [32.417311523031195]
  We propose learning representations from a set of implied, visually grounded expressions between image and text. We show that state-of-the-art multimodal learning models can be further improved by leveraging automatically harvested structural relations.
  arXiv  Detail & Related papers  (2020-10-06T18:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.