mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval

• URL: http://arxiv.org/abs/2604.17054v1
• Date: Sat, 18 Apr 2026 16:23:05 GMT
• Title: mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval
• Abstract summary: We propose a training-free, instruction-guided multimodal embedding framework.<n>We control the direction of embeddings through modality-specific instructions and structural cues.<n>Using a repurposed VGBench, we build the first text-to-SVG retrieval benchmark.
• Score: 23.372578915400613
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scalable Vector Graphics (SVGs) function both as visual images and as structured code that encode rich geometric and layout information, yet most methods rasterize them and discard this symbolic organization. At the same time, recent sentence embedding methods produce strong text representations but do not naturally extend to visual or structured modalities. We propose a training-free, instruction-guided multimodal embedding framework that uses a Multimodal Large Language Model (MLLM) to map text, raster images, and SVG code into an aligned embedding space. We control the direction of embeddings through modality-specific instructions and structural SVG cues, eliminating the need for learned projection heads or contrastive training. Our method has two key components: (1) Multimodal Explicit One-word Limitation (mEOL), which instructs the MLLM to summarize any multimodal input into a single token whose hidden state serves as a compact semantic embedding. (2) A semantic SVG rewriting module that assigns meaningful identifiers and simplifies nested SVG elements through visual reasoning over the rendered image, exposing geometric and relational cues hidden in raw code. Using a repurposed VGBench, we build the first text-to-SVG retrieval benchmark and show that our training-free embeddings outperform encoder-based and training-based multimodal baselines. These results highlight prompt-level control as an effective alternative to parameter-level training for structure-aware multimodal retrieval. Project page: https://scene-the-ella.github.io/meol/

Related papers

• A Scalable Vector Graphics Latent Space [7.25832782390665]
  We introduce SLS, a Transformer-based autoencoder that learns compact dense representations of individual SVG paths.<n> SLS learns fixed-size latent representations that jointly capture structure and appearance, and can be decoded back into valid, style-consistent paths with high fidelity.<n>We demonstrate that SLSs across diverse tasks reducing their FLOPs by over 150 times compared to token-based approaches.
  arXiv  Detail & Related papers  (2026-08-22T10:12:14Z)
• Multimodal OCR: Parse Anything from Documents [72.69545534962234]
  dots.mocr treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets.<n>It reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction.<n>It supports end-to-end training over heterogeneous document elements.
  arXiv  Detail & Related papers  (2026-03-13T14:42:21Z)
• DuetSVG: Unified Multimodal SVG Generation with Internal Visual Guidance [48.98604326855894]
  We introduce DuetSVG, a unified multimodal model that jointly generates image tokens and corresponding SVG tokens in an end-to-end manner.<n>At inference, we apply a novel test-time scaling strategy that leverages the model's native visual predictions as guidance to improve SVG decoding quality.
  arXiv  Detail & Related papers  (2025-12-11T18:23:03Z)
• VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation [51.95090758710288]
  We advocate SVG code as a compact, interpretable, and executable visual representation.<n>We introduce VCode, a benchmark that reframes multimodal understanding as code generation.<n>VCode covers three domains - general commonsense (MM-Vet), professional disciplines (MMMU), and visual-centric perception (CV-Bench)
  arXiv  Detail & Related papers  (2025-11-04T18:00:18Z)
• SVGThinker: Instruction-Aligned and Reasoning-Driven Text-to-SVG Generation [47.390332111383294]
  We present SVGThinker, a reasoning-driven framework that aligns the production of SVG code with the visualization process.<n>Our pipeline first renders each primitive in sequence and uses a multimodal model to annotate the image and code.<n> Experiments against state-of-the-art baselines show that SVGThinker produces more stable, editable, and higher-quality SVGs.
  arXiv  Detail & Related papers  (2025-09-29T05:25:00Z)
• SVGen: Interpretable Vector Graphics Generation with Large Language Models [61.62816031675714]
  We introduce SVG-1M, a large-scale dataset of high-quality SVGs paired with natural language descriptions.<n>We create well-aligned Text to SVG training pairs, including a subset with Chain of Thought annotations for enhanced semantic guidance.<n>Based on this dataset, we propose SVGen, an end-to-end model that generates SVG code from natural language inputs.
  arXiv  Detail & Related papers  (2025-08-06T15:00:24Z)
• Empowering LLMs to Understand and Generate Complex Vector Graphics [30.21003939248769]
  Large Language Models (LLMs) encode partial knowledge of vector graphics from web pages during training.<n>Recent findings suggest that semantically ambiguous and tokenized representations within LLMs may result in hallucinations in vector primitive predictions.<n>We present LLM4SVG, an initial yet substantial step toward bridging this gap by enabling LLMs to better understand and generate vector graphics.
  arXiv  Detail & Related papers  (2024-12-15T07:49:31Z)
• Visually Descriptive Language Model for Vector Graphics Reasoning [76.42082386029206]
  We propose the Visually Descriptive Language Model (VDLM) to bridge the gap between low-level visual perception and high-level language reasoning.<n>We show that VDLM significantly improves state-of-the-art LMMs like GPT-4o on various multimodal perception and reasoning tasks.
  arXiv  Detail & Related papers  (2024-04-09T17:30:18Z)
• ImageBind-LLM: Multi-modality Instruction Tuning [70.05191504511188]
  ImageBind-LLM is a multi-modality instruction tuning method of large language models (LLMs) via ImageBind. It can respond to audio, 3D point clouds, video, and their embedding-space arithmetic by only image-text alignment training.
  arXiv  Detail & Related papers  (2023-09-07T17:59:45Z)
• Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understanding [46.042197741423365]
  Large language models (LLMs) have made significant advancements in natural language understanding. This work investigates if it is possible for the LLM to understand images as well.
  arXiv  Detail & Related papers  (2023-06-09T17:57:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.