Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks

• URL: http://arxiv.org/abs/2405.10122v1
• Date: Thu, 16 May 2024 14:22:20 GMT
• Title: Generating Coherent Sequences of Visual Illustrations for Real-World Manual Tasks
• Authors: João Bordalo, Vasco Ramos, Rodrigo Valério, Diogo Glória-Silva, Yonatan Bitton, Michal Yarom, Idan Szpektor, Joao Magalhaes,
• Abstract summary: We propose an approach for generating consistent image sequences. Experiments show that the proposed approach is preferred by humans in 46.6% of the cases.
• Score: 8.346173383939961
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multistep instructions, such as recipes and how-to guides, greatly benefit from visual aids, such as a series of images that accompany the instruction steps. While Large Language Models (LLMs) have become adept at generating coherent textual steps, Large Vision/Language Models (LVLMs) are less capable of generating accompanying image sequences. The most challenging aspect is that each generated image needs to adhere to the relevant textual step instruction, as well as be visually consistent with earlier images in the sequence. To address this problem, we propose an approach for generating consistent image sequences, which integrates a Latent Diffusion Model (LDM) with an LLM to transform the sequence into a caption to maintain the semantic coherence of the sequence. In addition, to maintain the visual coherence of the image sequence, we introduce a copy mechanism to initialise reverse diffusion processes with a latent vector iteration from a previously generated image from a relevant step. Both strategies will condition the reverse diffusion process on the sequence of instruction steps and tie the contents of the current image to previous instruction steps and corresponding images. Experiments show that the proposed approach is preferred by humans in 46.6% of the cases against 26.6% for the second best method. In addition, automatic metrics showed that the proposed method maintains semantic coherence and visual consistency across steps in both domains.

Related papers

• Scale Your Instructions: Enhance the Instruction-Following Fidelity of Unified Image Generation Model by Self-Adaptive Attention Scaling [54.54513714247062]
  Recent advancements in unified image generation models, such as OmniGen, have enabled the handling of diverse image generation and editing tasks within a single framework.<n>We found that it suffers from text instruction neglect, especially when the text instruction contains multiple sub-instructions.<n>We propose Self-Adaptive Attention Scaling to dynamically scale the attention activation for each sub-instruction.
  arXiv  Detail & Related papers  (2025-07-22T05:25:38Z)
• Latent Beam Diffusion Models for Decoding Image Sequences [11.865234147230616]
  Existing methods generate each image independently, leading to disjointed narratives. We introduce a novel beam search strategy for latent space exploration, enabling conditional generation of full image sequences. By bridging advances in search optimization and latent space refinement, this work sets a new standard for structured image sequence generation.
  arXiv  Detail & Related papers  (2025-03-26T11:01:10Z)
• Autoregressive Image Generation with Randomized Parallel Decoding [23.714192351237628]
  ARPG is a novel visual autoregressive model that enables randomized parallel generation. Our approach attains an FID of 1.94 with only 64 sampling steps, achieving over a 20-fold increase in throughput.
  arXiv  Detail & Related papers  (2025-03-13T17:19:51Z)
• ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions [57.304601070962086]
  The goal of this work is to generate step-by-step visual instructions in the form of a sequence of images, given an input image. Part of the challenge stems from the lack of large-scale training data for this problem. We introduce an automatic approach for collecting large step-by-step visual instruction training data from instructional videos. Second, we develop and train ShowHowTo, a video diffusion model capable of generating step-by-step visual instructions consistent with the provided input image.
  arXiv  Detail & Related papers  (2024-12-02T21:40:17Z)
• ImPoster: Text and Frequency Guidance for Subject Driven Action Personalization using Diffusion Models [55.43801602995778]
  We present ImPoster, a novel algorithm for generating a target image of a'source' subject performing a 'driving' action. Our approach is completely unsupervised and does not require any access to additional annotations like keypoints or pose.
  arXiv  Detail & Related papers  (2024-09-24T01:25:19Z)
• Semantic Alignment for Multimodal Large Language Models [72.10272479476161]
  We introduce Semantic Alignment for Multi-modal large language models (SAM) By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis. By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis.
  arXiv  Detail & Related papers  (2024-08-23T06:48:46Z)
• Coherent Zero-Shot Visual Instruction Generation [15.0521272616551]
  This paper introduces a simple, training-free framework to tackle the issues of generating visual instructions. Our approach systematically integrates text comprehension and image generation to ensure visual instructions are visually appealing. Our experiments show that our approach can visualize coherent and visually pleasing instructions.
  arXiv  Detail & Related papers  (2024-06-06T17:59:44Z)
• TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation [44.740794326596664]
  TheaterGen is a training-free framework that integrates large language models (LLMs) and text-to-image (T2I) models. Within this framework, LLMs, acting as "Screenwriter", engage in multi-turn interaction, generating and managing a standardized prompt book. With the effective management of prompt books and character images, TheaterGen significantly improves semantic and contextual consistency in synthesized images.
  arXiv  Detail & Related papers  (2024-04-29T17:58:14Z)
• Texture, Shape, Order, and Relation Matter: A New Transformer Design for Sequential DeepFake Detection [81.04969838138634]
  Sequential DeepFake detection is an emerging task that predicts the manipulation sequence in order.<n>This paper describes a new Transformer design, called TSOM, by exploring three perspectives: Texture, Shape, and Order of Manipulations.<n>We conduct extensive experiments in comparison with several state-of-the-art methods, demonstrating the superiority of our method.
  arXiv  Detail & Related papers  (2024-04-22T04:47:52Z)
• Masked Generative Story Transformer with Character Guidance and Caption Augmentation [2.1392064955842023]
  Story visualization is a challenging generative vision task, that requires both visual quality and consistency between different frames in generated image sequences. Previous approaches either employ some kind of memory mechanism to maintain context throughout an auto-regressive generation of the image sequence, or model the generation of the characters and their background separately. We propose a completely parallel transformer-based approach, relying on Cross-Attention with past and future captions to achieve consistency.
  arXiv  Detail & Related papers  (2024-03-13T13:10:20Z)
• Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance [51.188396199083336]
  We present an approach that guides the reverse process of diffusion sampling by applying asymmetric gradient guidance. Our model's adaptability allows it to be implemented with both image-fusion and latent-dif models. Experiments show that our method outperforms various state-of-the-art models in image translation tasks.
  arXiv  Detail & Related papers  (2023-06-07T12:56:56Z)
• RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment [112.45442468794658]
  We propose a two-stage coarse-to-fine semantic re-alignment method, named RealignDiff. In the coarse semantic re-alignment phase, a novel caption reward is proposed to evaluate the semantic discrepancy between the generated image caption and the given text prompt. The fine semantic re-alignment stage employs a local dense caption generation module and a re-weighting attention modulation module to refine the previously generated images from a local semantic view.
  arXiv  Detail & Related papers  (2023-05-31T06:59:21Z)
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis [15.006676130258372]
  Autoregressive models incorporate context in a linear 1D order by attending only to previously synthesized image patches above or to the left. We propose a coarse-to-fine hierarchy of context by combining the autoregressive formulation with a multinomial diffusion process. Our approach can take unrestricted, user-provided masks into account to perform local image editing.
  arXiv  Detail & Related papers  (2021-08-19T17:50:07Z)
• Cycle-Consistent Inverse GAN for Text-to-Image Synthesis [101.97397967958722]
  We propose a novel unified framework of Cycle-consistent Inverse GAN for both text-to-image generation and text-guided image manipulation tasks. We learn a GAN inversion model to convert the images back to the GAN latent space and obtain the inverted latent codes for each image. In the text-guided optimization module, we generate images with the desired semantic attributes by optimizing the inverted latent codes.
  arXiv  Detail & Related papers  (2021-08-03T08:38:16Z)
• Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene [10.822477939237459]
  We propose contrastive masked language modeling (CMLM) for post-training to integrate both token-level and sequence-level contrastive learnings. CMLM surpasses several recent post-training methods in few-shot settings without the need for data augmentation.
  arXiv  Detail & Related papers  (2021-06-04T08:17:48Z)
• Towards Open-World Text-Guided Face Image Generation and Manipulation [52.83401421019309]
  We propose a unified framework for both face image generation and manipulation. Our method supports open-world scenarios, including both image and text, without any re-training, fine-tuning, or post-processing.
  arXiv  Detail & Related papers  (2021-04-18T16:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.