Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models

• URL: http://arxiv.org/abs/2606.19944v1
• Date: Thu, 18 Jun 2026 08:40:41 GMT
• Title: Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models
• Abstract summary: We introduce Timage, a paradigm that recasts multimodal understanding as an alignment problem solved at the input.<n>The study positions deliberate input reconstruction as a powerful, architecture-neutral lever for strengthening multimodal reasoning.
• Score: 22.803631190636313
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) often lose track of the right image regions during fine-grained spatial reasoning, because a textual query rarely carries any explicit geometric anchor into the pixel domain. Prevailing remedies either rewire the model's weights or pad the prompt with verbose instructions, yet neither reliably pins the language to the correct visual coordinates without eroding the backbone's general competence. We introduce Timage, a paradigm that recasts multimodal understanding as an alignment problem solved at the input: the query is drawn, as a typeset overlay, onto the image itself. The placement and appearance of this overlay are produced by a Constrained Schrödinger Bridge (cSB), an entropic optimal-transport sampler that factorizes layout synthesis into two coupled stochastic stages. The first stage, Region Search, transports noise toward query-aligned image zones while obeying a hard occlusion barrier that protects salient foreground content; the second stage, Appearance Shaping, sizes the glyphs through an ``ink-budget'' regularizer so that the rendered text stays legible and visually balanced. The resulting overlay behaves as an explicit attention beacon that channels the model's focus along spatial semantics. On the VMCBench suite, Timage paired with a modest 7B backbone clearly overtakes far larger proprietary systems as well as parameter-tuned baselines. The study positions deliberate input reconstruction as a powerful, architecture-neutral lever for strengthening multimodal reasoning.

Related papers

• One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting [51.09550363815034]
  Scene text spotting requires high-precision alignment between textual recognition and spatial localization.<n>We propose Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token.<n>We show that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs.
  arXiv  Detail & Related papers  (2026-07-30T09:17:48Z)
• Imagine Before You Draw: Visual Prompt Engineering for Image Generation [67.81347924426714]
  We propose Visual Prompt Engineering (VPE), which can be seamlessly integrated into internal frameworks.<n>We validate VPE across class-conditional generation, text-to-image generation, and image editing.<n>Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration, achieve substantially better editing preservation.
  arXiv  Detail & Related papers  (2026-06-03T05:01:36Z)
• Envisioning Beyond the Few: Disentangled Semantics and Primitives for Few-Shot Atypical Layout-to-Image Generation [11.600113323546404]
  We propose a representation-driven framework that disentangles semantics from primitives for robust few-shot adaptation.<n>Semantic Anchoring aggregates categorical semantics into anchors for stable identity, while Primitive Imbuing models recomposable primitives for robust local detail modeling.<n>Experiments demonstrate consistent improvements in the 5-shot regime over state-of-the-art L2I methods in both visual fidelity and alignment across diverse domains.
  arXiv  Detail & Related papers  (2026-05-29T13:00:00Z)
• Semantic-Structural Alignment for Generative Pictorial Charts [59.45629259524998]
  We present a generative framework for the automated synthesis of pictorial charts.<n>We frame the problem as a dual-conditioned generation task guided by two parallel external control signals.<n>Our method produces pictorial charts that are both artistically compelling and structurally consistent.
  arXiv  Detail & Related papers  (2026-05-05T05:20:46Z)
• Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation [0.3437656066916039]
  Referring image segmentation aims to produce a pixel-level mask for the image region described by a natural-language expression.<n>We propose SERA, a Spatio-Semantic Expert Routing Architecture for referring image segmentation.<n> SERA introduces lightweight, expression-aware expert refinement at two complementary stages within a vision-language framework.
  arXiv  Detail & Related papers  (2026-03-13T00:37:20Z)
• Unpaired Image-to-Image Translation via a Self-Supervised Semantic Bridge [59.247871132422006]
  Adversarial diffusion and diffusion-inversion methods have advanced unpaired image-to-image translation, but each faces key limitations.<n>We propose the Self-Supervised Semantic Bridge ( SSB), a versatile framework that integrates external semantic priors into diffusion bridge models.<n>Our key idea is to leverage self-supervised visual encoders to learn representations that are invariant to appearance changes but capture geometric structure.
  arXiv  Detail & Related papers  (2026-02-18T18:05:00Z)
• Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models [84.78794648147608]
  A persistent geometric anomaly, the Modality Gap, remains.<n>Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions.<n>We propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap into stable biases and anisotropic residuals.<n>We then introduce ReAlign, a training-free modality alignment strategy.
  arXiv  Detail & Related papers  (2026-02-02T13:59:39Z)
• How Modality Shapes Perception and Reasoning: A Study of Error Propagation in ARC-AGI [7.226300346775942]
  ARC-AGI and ARC-AGI-2 measure generalization-through-composition on small color-quantized grids.<n>Recent instruction-first systems translate grids into concise natural-language or DSL rules executed in generate-execute-select loops.
  arXiv  Detail & Related papers  (2025-11-11T19:06:41Z)
• A Large-scale Interpretable Multi-modality Benchmark for Facial Image Forgery Localization [22.725542948364357]
  We argue that the basic binary forgery mask is inadequate for explaining model predictions.<n>In this study, we generate salient region-focused interpretation for the forgery images.<n>We develop ForgeryTalker, an architecture designed for concurrent forgery localization and interpretation.
  arXiv  Detail & Related papers  (2024-12-27T15:23:39Z)
• Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation [81.45400849638347]
  In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image containing translations in target language. In this paper, we propose an end-to-end IIMT model consisting of four modules. Our model achieves competitive performance compared to cascaded models with only 70.9% of parameters, and significantly outperforms the pixel-level end-to-end IIMT model.
  arXiv  Detail & Related papers  (2024-07-03T08:15:39Z)
• Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid [102.24539566851809]
  Restoring reasonable and realistic content for arbitrary missing regions in images is an important yet challenging task. Recent image inpainting models have made significant progress in generating vivid visual details, but they can still lead to texture blurring or structural distortions. We propose the Semantic Pyramid Network (SPN) motivated by the idea that learning multi-scale semantic priors can greatly benefit the recovery of locally missing content in images.
  arXiv  Detail & Related papers  (2021-12-08T04:33:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.