Agentic Visual Generation: From Generative Models to Agentic Control

• URL: http://arxiv.org/abs/2609.06758v1
• Date: Sun, 06 Sep 2026 17:59:24 GMT
• Title: Agentic Visual Generation: From Generative Models to Agentic Control
• Abstract summary: In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors.<n>Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity.<n>We organize the field according to what the controller can directly control in the generation process.
• Score: 73.3572965008302
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.

Related papers

• GPC: Large-Scale Generative Pretraining for Transferable Motor Control [13.694660962373703]
  We introduce Generative Pretrained Controllers, which leverage tokenization and next-token modeling to create reusable generative controllers.<n>Our framework utilizes end-to-end reinforcement learning to jointly optimize a "motion vocabulary"<n>The generative controller exhibits a variety of natural emergent behaviors, such as responsive behaviors to perturbations and recovery behaviors after falling.
  arXiv  Detail & Related papers  (2026-06-28T02:05:11Z)
• Iterative Critique-and-Routing Controller for Multi-Agent Systems with Heterogeneous LLMs [31.34021188099265]
  Multi-agent large language model (LLM) systems often rely on a controller to coordinate a pool of heterogeneous models.<n>We propose a critique-and-routing controller that casts multi-agent coordination as a sequential decision problem.
  arXiv  Detail & Related papers  (2026-05-09T04:51:42Z)
• Gated Condition Injection without Multimodal Attention: Towards Controllable Linear-Attention Transformers [69.60693631859411]
  This paper explores controllable diffusion models built upon linear attention architectures.<n>We propose a novel controllable diffusion framework tailored for linear attention backbones like SANA.<n>Our approach achieves state-of-the-art controllable generation performance based on linear-attention models.
  arXiv  Detail & Related papers  (2026-03-29T12:31:34Z)
• GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoning [54.42973725693]
  We introduce GenAgent, unifying visual understanding and generation through an agentic multimodal model.<n>GenAgent significantly boosts base generator(FLUX.1-dev) performance on GenEval++ and WISE.<n>Our framework demonstrates three key properties: 1) cross-tool generalization to generators with varying capabilities, 2) test-time scaling with consistent improvements across interaction rounds, and 3) task-adaptive reasoning that automatically adjusts to different tasks.
  arXiv  Detail & Related papers  (2026-01-26T14:49:04Z)
• GenCtrl -- A Formal Controllability Toolkit for Generative Models [10.810813965290707]
  We propose a novel algorithm to estimate the controllable sets of models in a dialogue setting.<n>We show that model controllability is surprisingly fragile and highly dependent on the experimental setting.
  arXiv  Detail & Related papers  (2026-01-09T08:50:02Z)
• SCALAR: Scale-wise Controllable Visual Autoregressive Learning [15.775596699630633]
  We present SCALAR, a controllable generation method based on Visual Autoregressive ( VAR)<n>We leverage a pretrained image encoder to extract semantic control signal encodings, which are projected into scale-specific representations and injected into the corresponding layers of the VAR backbone.<n>Building on SCALAR, we develop SCALAR-Uni, a unified extension that aligns multiple control modalities into a shared latent space, supporting flexible multi-conditional guidance in a single model.
  arXiv  Detail & Related papers  (2025-07-26T13:23:08Z)
• A Practical Investigation of Spatially-Controlled Image Generation with Transformers [23.373853892284227]
  We aim to provide clear takeaways across generation paradigms for spatially-controlled generation.<n>We perform experiments on ImageNet across diffusion-based/flow-based and autoregressive (AR) models.<n>We re-clarify the motivation of adapter-based approaches, demonstrating that they mitigate "forgetting" and maintain generation quality when trained on limited downstream data.
  arXiv  Detail & Related papers  (2025-07-21T15:33:49Z)
• Controlling Large Language Models Through Concept Activation Vectors [30.348768212571255]
  We propose Generation with Concept Activation Vector (GCAV), a lightweight model control framework.<n>GCAV ensures accurate control without requiring resource-extensive fine-tuning.<n>Our framework achieves state-of-the-art performance with granular control, allowing for fine-grained adjustments of both the steering layers and the steering magnitudes for individual samples.
  arXiv  Detail & Related papers  (2025-01-10T07:41:48Z)
• CAR: Controllable Autoregressive Modeling for Visual Generation [100.33455832783416]
  Controllable AutoRegressive Modeling (CAR) is a novel, plug-and-play framework that integrates conditional control into multi-scale latent variable modeling. CAR progressively refines and captures control representations, which are injected into each autoregressive step of the pre-trained model to guide the generation process. Our approach demonstrates excellent controllability across various types of conditions and delivers higher image quality compared to previous methods.
  arXiv  Detail & Related papers  (2024-10-07T00:55:42Z)
• PerLDiff: Controllable Street View Synthesis Using Perspective-Layout Diffusion Models [55.080748327139176]
  We introduce PerLDiff, a novel method for effective street view image generation that fully leverages perspective 3D geometric information.<n>PerLDiff employs 3D geometric priors to guide the generation of street view images with precise object-level control within the network learning process.<n> Empirical results justify that our PerLDiff markedly enhances the precision of controllable generation on the NuScenes and KITTI datasets.
  arXiv  Detail & Related papers  (2024-07-08T16:46:47Z)
• Unsupervised Controllable Generation with Self-Training [90.04287577605723]
  controllable generation with GANs remains a challenging research problem. We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-17T21:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.