WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation

• URL: http://arxiv.org/abs/2511.11434v1
• Date: Fri, 14 Nov 2025 16:02:38 GMT
• Title: WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation
• Authors: Wei Chow, Jiachun Pan, Yongyuan Liang, Mingze Zhou, Xue Song, Liyu Jia, Saining Zhang, Siliang Tang, Juncheng Li, Fengda Zhang, Weijia Wu, Hanwang Zhang, Tat-Seng Chua,
• Abstract summary: We present WEAVE, the first suite for in-context interleaved cross-modality comprehension and generation.<n>WeAVE-100k is a large-scale dataset of 100K interleaved samples spanning over 370K dialogue turns and 500K images.<n>WeAVEBench is a human-annotated benchmark with 100 tasks based on 480 images.
• Score: 98.47375190901447
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in unified multimodal models (UMMs) have enabled impressive progress in visual comprehension and generation. However, existing datasets and benchmarks focus primarily on single-turn interactions, failing to capture the multi-turn, context-dependent nature of real-world image creation and editing. To address this gap, we present WEAVE, the first suite for in-context interleaved cross-modality comprehension and generation. Our suite consists of two complementary parts. WEAVE-100k is a large-scale dataset of 100K interleaved samples spanning over 370K dialogue turns and 500K images, covering comprehension, editing, and generation tasks that require reasoning over historical context. WEAVEBench is a human-annotated benchmark with 100 tasks based on 480 images, featuring a hybrid VLM judger evaluation framework based on both the reference image and the combination of the original image with editing instructions that assesses models' abilities in multi-turn generation, visual memory, and world-knowledge reasoning across diverse domains. Experiments demonstrate that training on WEAVE-100k enables vision comprehension, image editing, and comprehension-generation collaboration capabilities. Furthermore, it facilitates UMMs to develop emergent visual-memory capabilities, while extensive evaluations on WEAVEBench expose the persistent limitations and challenges of current approaches in multi-turn, context-aware image generation and editing. We believe WEAVE provides a view and foundation for studying in-context interleaved comprehension and generation for multi-modal community.

Related papers

• MICON-Bench: Benchmarking and Enhancing Multi-Image Context Image Generation in Unified Multimodal Models [89.89575486159795]
  We introduce textbfMICON-Bench, a benchmark for multi-image context generation.<n>We propose an MLLM-driven Evaluation-by-Checkpoint framework for automatic verification of semantic and visual consistency.<n>We also present textbfDynamic Attention Rebalancing (DAR), a training-free, plug-and-play mechanism that dynamically adjusts attention during inference to enhance coherence and reduce hallucinations.
  arXiv  Detail & Related papers  (2026-02-23T04:32:52Z)
• FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback [92.67587639164908]
  We present FronTalk, a benchmark for front-end code generation with multi-modal feedback.<n>We focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues.<n> Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature.
  arXiv  Detail & Related papers  (2025-12-05T23:28:09Z)
• Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learning [92.57052246970254]
  We propose to enable the collaborative co-evolution of visual comprehension and generation.<n>We introduce a two-stage training approach: supervised fine-tuning teaches the MLLM with the foundational ability to generate genuine CoT.<n>We unlock the Aha moment in visual generation, advancing MLLMs from text-to-image tasks to unified image generation.
  arXiv  Detail & Related papers  (2025-06-02T09:39:28Z)
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation [59.53678957969471]
  Multimodal Large Language Models (MLLMs) have made significant strides in visual understanding and generation tasks.<n> generating interleaved image-text content remains a challenge.<n>OpenING is a benchmark comprising 5,400 high-quality human-annotated instances across 56 real-world tasks.<n>IntJudge is a judge model for evaluating open-ended multimodal generation methods.
  arXiv  Detail & Related papers  (2024-11-27T16:39:04Z)
• Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment [53.45813302866466]
  We present ISG, a comprehensive evaluation framework for interleaved text-and-image generation.<n>ISG evaluates responses on four levels of granularity: holistic, structural, block-level, and image-specific.<n>In conjunction with ISG, we introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8 categories and 21 subcategories.
  arXiv  Detail & Related papers  (2024-11-26T07:55:57Z)
• Open-World Human-Object Interaction Detection via Multi-modal Prompts [26.355054079885463]
  MP-HOI is a powerful Multi-modal Prompt-based HOI detector designed to leverage both textual descriptions for open-set generalization and visual exemplars for handling high ambiguity in descriptions. MP-HOI could serve as a generalist HOI detector, surpassing the HOI vocabulary of existing expert models by more than 30 times.
  arXiv  Detail & Related papers  (2024-06-11T13:01:45Z)
• JourneyDB: A Benchmark for Generative Image Understanding [89.02046606392382]
  We introduce a comprehensive dataset, referred to as JourneyDB, that caters to the domain of generative images. Our meticulously curated dataset comprises 4 million distinct and high-quality generated images. On our dataset, we have devised four benchmarks to assess the performance of generated image comprehension.
  arXiv  Detail & Related papers  (2023-07-03T02:39:08Z)
• Generative Modeling for Multi-task Visual Learning [40.96212750592383]
  We consider a novel problem of learning a shared generative model that is useful across various visual perception tasks. We propose a general multi-task oriented generative modeling framework, by coupling a discriminative multi-task network with a generative network. Our framework consistently outperforms state-of-the-art multi-task approaches.
  arXiv  Detail & Related papers  (2021-06-25T03:42:59Z)
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data [13.68491474904529]
  We propose Text-enhanced Visual Deep InfoMax (TVDIM) to learn better visual representations. Our core idea of self-supervised learning is to maximize the mutual information between features extracted from multiple views. TVDIM significantly outperforms previous visual self-supervised methods when processing the same set of images.
  arXiv  Detail & Related papers  (2021-06-03T12:36:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.