Vision-Language Binding in In-Context Image Generation

• URL: http://arxiv.org/abs/2605.24624v1
• Date: Sat, 23 May 2026 15:30:19 GMT
• Title: Vision-Language Binding in In-Context Image Generation
• Abstract summary: In-context image generation models such as FLUX.2 take a text prompt and an optional reference image as visual conditioning for the output.<n>We show that an implicit cross-modal binding emerges between the text tokens and the reference image.<n>We further localize the reference-text binding to the padding tokens of the text sequence.
• Score: 34.419605761529404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context image generation models such as FLUX.2 take a text prompt and an optional reference image as visual conditioning for the output. Internally, all three inputs -- text, reference image, and the noise tokens -- are concatenated and processed through a single attention stream, where all tokens can attend to one another. This leaves open how reference information flows through the model to produce the output image. We show that an implicit cross-modal binding emerges between the text tokens and the reference image: the text tokens absorb visual reference content during the forward pass, and that absorbed content causally influences the generated output. We surface this binding with three causal interventions on FLUX.2: T2I Lens, which decodes intermediate text-token activations through a text-to-image path; Attention Knockout, which severs specific attention edges; and I2I-to-I2I Patching, which copies text token activations between editing runs. Across 2,875 editing tasks on various images, including SUN397 and DreamBench++ datasets and images collected online, we observe a consistent division of labor: properties of the reference image, like color, style, and scene setting, are first written into the text tokens, which carry them to the generated image; pixel-exact properties like a specific face or instance identity bypass the text tokens and flow directly from reference to image through image-to-image attention. We further localize the reference-text binding to the padding tokens of the text sequence. These results show that text tokens in a multimodal DiT are not just prompt holders, but a structured channel for reference image content. More broadly, they suggest that even in unified-attention multimodal generative models, token modality structures how conditioning information is represented and routed across the network.

Related papers

• Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers [16.996490884328455]
  Text-to-image diffusion transformers (DiTs) generate images through joint attention.<n>In large-scale DiTs, the conditioning input contains not only the user prompt but also chat-template tokens.<n>We introduce a causal interpretability framework to probe how these tokens participate in the denoising computation.
  arXiv  Detail & Related papers  (2026-07-21T14:29:59Z)
• Text-to-Image Models Need Less from Text Encoders Than You Think [37.573078297644415]
  We show that text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order.<n>This demonstrates that, contrary to common belief, text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order.
  arXiv  Detail & Related papers  (2026-06-02T14:37:57Z)
• UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation [51.31795451147935]
  We present a unified generative model that supports visual understanding and visual generation within a single pixel-to-pixel diffusion framework.<n>Our goal is to achieve unification along three axes: the model, the tasks, and the representations.<n> Experiments on text-to-image synthesis and image-to-text understanding demonstrate strong cross-modal alignment.
  arXiv  Detail & Related papers  (2025-11-21T03:02:10Z)
• Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models [35.85433370296494]
  We investigate how semantic information is distributed across token representations in a text-to-image model.<n>We find information is usually concentrated in only one or two of the item's tokens.<n>In some cases, items do influence each other's representation, often leading to misinterpretations.
  arXiv  Detail & Related papers  (2025-04-01T19:17:44Z)
• Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models [64.52046218688295]
  Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process.<n>We conduct the first in-depth analysis of the role padding tokens play in T2I models.<n>Our findings reveal three distinct scenarios: padding tokens may affect the model's output during text encoding, during the diffusion process, or be effectively ignored.
  arXiv  Detail & Related papers  (2025-01-12T08:36:38Z)
• Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control [58.37323932401379]
  Current diffusion models create images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. We propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.
  arXiv  Detail & Related papers  (2024-04-21T20:26:46Z)
• Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval [53.89454443114146]
  We study the zero-shot Composed Image Retrieval (ZS-CIR) task, which is to retrieve the target image given a reference image and a description without training on the triplet datasets. Previous works generate pseudo-word tokens by projecting the reference image features to the text embedding space. We propose a Knowledge-Enhanced Dual-stream zero-shot composed image retrieval framework (KEDs) KEDs implicitly models the attributes of the reference images by incorporating a database.
  arXiv  Detail & Related papers  (2024-03-24T04:23:56Z)
• Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models [8.250234707160793]
  Recent text-to-image (T2I) diffusion models show outstanding performance in generating high-quality images conditioned on textual prompts. They fail to semantically align the generated images with the prompts due to their limited compositional capabilities. We propose a novel attention mask control strategy based on predicted object boxes to address these issues.
  arXiv  Detail & Related papers  (2023-05-23T10:49:22Z)
• Character-Centric Story Visualization via Visual Planning and Token Alignment [53.44760407148918]
  Story visualization advances the traditional text-to-image generation by enabling multiple image generation based on a complete story. Key challenge of consistent story visualization is to preserve characters that are essential in stories. We propose to adapt a recent work that augments Vector-Quantized Variational Autoencoders with a text-tovisual-token architecture.
  arXiv  Detail & Related papers  (2022-10-16T06:50:39Z)
• CM3: A Causal Masked Multimodal Model of the Internet [86.32652030161374]
  We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents. We train causally masked language-image models on large-scale web and Wikipedia articles. CM3 models can generate rich structured, multi-modal outputs while conditioning on arbitrary masked document contexts.
  arXiv  Detail & Related papers  (2022-01-19T10:45:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.