Latent Action Control for Reasoning-Guided Unified Image Generation

• URL: http://arxiv.org/abs/2605.16961v1
• Date: Sat, 16 May 2026 12:23:20 GMT
• Title: Latent Action Control for Reasoning-Guided Unified Image Generation
• Abstract summary: We propose Latent Action Control (LAC), which makes reasoning actionable by representing it as hidden continuous actions inside a unified generator.<n>We show that LAC consistently improves compositional and knowledge-grounded generation across GenEval, WISE, and T2I-CompBench.
• Score: 15.169749682149314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unified multimodal models can encode visual understanding and image generation within a shared backbone, yet understanding does not automatically translate into control: models may infer objects, relations, or knowledge cues but fail to instantiate them in the generated image. We propose Latent Action Control (LAC), which makes reasoning actionable by representing it as hidden continuous actions inside a unified generator. Given a prompt, LAC rolls out a role-structured latent trajectory for planning, internal visual drafting, diagnosis, and refinement, and injects these actions into the hidden stream that conditions flow-based generation, without producing reasoning tokens or intermediate images. Since such action trajectories are unobserved, LAC learns them through prior-guided variational latent action alignment from training-only rendered semantic priors, draft image features, and supervised halting signals, followed by Latent-Flow GRPO to align the latent-to-image rollout with terminal visual feedback. This provides a control path from inferred relations, bindings, and knowledge cues to the generation process. Instantiated on BAGEL-7B-MoT, LAC consistently improves compositional and knowledge-grounded generation across GenEval, WISE, and T2I-CompBench, with the largest gains on spatial relations, attribute binding, and world-knowledge-sensitive prompts. Ablations and latent interventions show that the learned action trajectory is consumed by the generator, suggesting that unified generation benefits when understanding is not only encoded, but made actionable during generation.

Related papers

• Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction [11.257989394325671]
  Generation-guided training framework reinterprets visual generation as auxiliary supervision for representation learning.<n>Gas improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension.
  arXiv  Detail & Related papers  (2026-08-12T16:03:31Z)
• Collaborative Memory Augmentation for Generative Recommendation [49.76630658239287]
  We propose textbfOMEGA, a cOllaborative MEmory augmentation framework for Generative recommendAtion.<n>We first introduce a latent context compression method that utilizes learnable query tokens to distill sequential user behavior into compact representations.<n>These compressed representations are aggregated into a collaborative memory bank, serving as an explicit repository of global behavioral patterns.
  arXiv  Detail & Related papers  (2026-08-02T15:34:43Z)
• No Prompt, No Leaks: A Robust Generative Steganography Framework via Prompt-Free Diffusion [15.432428509145703]
  Generative image steganography synthesizes stego images directly from secret information to achieve inherent security advantages.<n>Latent Diffusion Models have emerged as a fundamental image steganography framework that modulates secret latent representations with text prompts.<n>We propose a prompt-free diffusion image steganography framework that integrates style semantic priors to control more robust and reliable stego image generation.
  arXiv  Detail & Related papers  (2026-06-30T09:51:34Z)
• Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance [108.46436073194546]
  We argue that effective planning should be local, visually grounded, internally generated, and directly aligned with action.<n>We propose Afford-VLA, a unified framework that internalizes task-conditioned affordance as an explicit visual planning interface within VLA models.
  arXiv  Detail & Related papers  (2026-05-22T20:43:47Z)
• ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both [55.182037225013836]
  ATLAS is a framework in which a single discrete 'word', termed as a functional token, serves both as an agentic operation and a latent visual reasoning unit.<n> ATLAS achieves superior performance on challenging benchmarks while maintaining clear interpretability.
  arXiv  Detail & Related papers  (2026-05-14T17:59:55Z)
• Steering Visual Generation in Unified Multimodal Models with Understanding Supervision [42.765106450407814]
  Unified multimodal models are envisioned to bridge the gap between understanding and generation.<n>We propose Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations.
  arXiv  Detail & Related papers  (2026-05-07T07:20:04Z)
• Self-Corrected Image Generation with Explainable Latent Rewards [55.29175717238288]
  We propose xLARD, a self-correcting framework that guides generation through Explainable LAtent RewarDs.<n>xLARD introduces a lightweight corrector that refines latent representations based on structured feedback from model-generated references.<n> Experiments show that xLARD improves semantic alignment and visual fidelity while maintaining generative priors.
  arXiv  Detail & Related papers  (2026-03-26T02:59:35Z)
• Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learning [43.715148652244864]
  KG-M3PO is a framework for multi-task robotic manipulation in partially observable settings that unifies Perception, Knowledge, and Policy.<n>The method augments egocentric vision with an online 3D scene graph that grounds open-vocabulary detections into a metric, relational representation.
  arXiv  Detail & Related papers  (2026-03-25T08:41:32Z)
• Going with the Flow: Koopman Behavioral Models as Implicit Planners for Visuo-Motor Dexterity [1.6865201067073452]
  We introduce Unified Behavioral Models (UBMs), a framework that learns to represent dexterous skills as coupled dynamical systems.<n>Koopman-UBM is a first instantiation of UBMs that learns a unified representation in which the joint flow of latent visual and proprioceptive features is governed by a structured linear system.
  arXiv  Detail & Related papers  (2026-02-07T07:18:00Z)
• Re-Align: Structured Reasoning-guided Alignment for In-Context Image Generation and Editing [38.240269144736224]
  Re-Align bridges the gap between understanding and generation through structured reasoning-guided alignment.<n>In-context image generation and editing (ICGE) enables users to specify visual concepts through interleaved image-text prompts.
  arXiv  Detail & Related papers  (2026-01-08T17:13:00Z)
• Chain-of-Image Generation: Toward Monitorable and Controllable Image Generation [7.987662261007762]
  Chain-of-Image Generation (CoIG) framework reframes image generation as a sequential, semantic process analogous to how humans create art.<n>Our experimental results indicate that CoIG substantially enhances quantitative monitorability while achieving competitive robustness compared to established baseline models.
  arXiv  Detail & Related papers  (2025-12-09T14:35:12Z)
• Unified Autoregressive Visual Generation and Understanding with Continuous Tokens [52.21981295470491]
  We present UniFluid, a unified autoregressive framework for joint visual generation and understanding.<n>Our unified autoregressive architecture processes multimodal image and text inputs, generating discrete tokens for text and continuous tokens for image.<n>We find though there is an inherent trade-off between the image generation and understanding task, a carefully tuned training recipe enables them to improve each other.
  arXiv  Detail & Related papers  (2025-03-17T17:58:30Z)
• Enhancing Graph Contrastive Learning with Reliable and Informative Augmentation for Recommendation [84.45144851024257]
  We propose a novel framework that aims to enhance graph contrastive learning by constructing contrastive views with stronger collaborative information via discrete codes.<n>The core idea is to map users and items into discrete codes rich in collaborative information for reliable and informative contrastive view generation.
  arXiv  Detail & Related papers  (2024-09-09T14:04:17Z)
• Unsupervised Controllable Generation with Self-Training [90.04287577605723]
  controllable generation with GANs remains a challenging research problem. We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-17T21:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.