StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation

• URL: http://arxiv.org/abs/2505.10292v1
• Date: Thu, 15 May 2025 13:42:14 GMT
• Title: StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation
• Authors: Daniel A. P. Oliveira, David Martins de Matos,
• Abstract summary: Visual storytelling systems struggle to maintain character identity across frames and link actions to appropriate subjects.<n>We propose StoryReasoning, a dataset containing 4,178 stories derived from 52,016 movie images.<n>We create Qwen Storyteller, which performs end-to-end object detection, re-identification, and landmark detection while maintaining consistent object references throughout the story.
• Score: 0.2455468619225742
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual storytelling systems struggle to maintain character identity across frames and link actions to appropriate subjects, frequently leading to referential hallucinations. These issues can be addressed through grounding of characters, objects, and other entities on the visual elements. We propose StoryReasoning, a dataset containing 4,178 stories derived from 52,016 movie images, with both structured scene analyses and grounded stories. Each story maintains character and object consistency across frames while explicitly modeling multi-frame relationships through structured tabular representations. Our approach features cross-frame object re-identification using visual similarity and face recognition, chain-of-thought reasoning for explicit narrative modeling, and a grounding scheme that links textual elements to visual entities across multiple frames. We establish baseline performance by fine-tuning Qwen2.5-VL 7B, creating Qwen Storyteller, which performs end-to-end object detection, re-identification, and landmark detection while maintaining consistent object references throughout the story. Evaluation demonstrates a reduction from 4.06 to 3.56 (-12.3%) hallucinations on average per story when compared to a non-fine-tuned model.

Related papers

• Entity Re-identification in Visual Storytelling via Contrastive Reinforcement Learning [0.2455468619225742]
  Visual storytelling systems struggle to maintain character and object identity across frames.<n>We propose a contrastive reinforcement learning approach that trains models to discriminate between coherent image sequences and stories from unrelated images.
  arXiv  Detail & Related papers  (2025-07-09T23:52:10Z)
• ViStoryBench: Comprehensive Benchmark Suite for Story Visualization [23.274981415638837]
  ViStoryBench is an evaluation benchmark for story visualization models.<n>It features stories with single and multiple protagonists to test models' ability to maintain character consistency.<n>It includes complex plots and intricate world-building to challenge models in generating accurate visuals.
  arXiv  Detail & Related papers  (2025-05-30T17:58:21Z)
• Structured Graph Representations for Visual Narrative Reasoning: A Hierarchical Framework for Comics [1.320904960556043]
  This paper presents a hierarchical knowledge graph framework for the structured understanding of visual narratives, focusing on comics.<n>It represents them through integrated knowledge graphs that capture semantic, spatial, and temporal relationships.<n>At the panel level, we construct multimodal graphs that link visual elements such as characters, objects, and actions with corresponding textual components, including dialogue and captions.
  arXiv  Detail & Related papers  (2025-04-14T14:42:19Z)
• Generating Visual Stories with Grounded and Coreferent Characters [63.07511918366848]
  We present the first model capable of predicting visual stories with consistently grounded and coreferent character mentions.<n>Our model is finetuned on a new dataset which we build on top of the widely used VIST benchmark.<n>We also propose new evaluation metrics to measure the richness of characters and coreference in stories.
  arXiv  Detail & Related papers  (2024-09-20T14:56:33Z)
• ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Context [50.572907418430155]
  ContextualStory is a framework designed to generate coherent story frames and extend frames for visual storytelling.<n>We introduce a Storyline Contextualizer to enrich context in storyline embedding, and a StoryFlow Adapter to measure scene changes between frames.<n>Experiments on PororoSV and FlintstonesSV datasets demonstrate that ContextualStory significantly outperforms existing SOTA methods in both story visualization and continuation.
  arXiv  Detail & Related papers  (2024-07-13T05:02:42Z)
• TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling [14.15543866199545]
  As a cross-modal task, visual storytelling aims to generate a story for an ordered image sequence automatically. We propose a novel method, Topic Aware Reinforcement Network for VIsual StoryTelling (TARN-VIST) In particular, we pre-extracted the topic information of stories from both visual and linguistic perspectives.
  arXiv  Detail & Related papers  (2024-03-18T08:01:23Z)
• Detecting and Grounding Important Characters in Visual Stories [18.870236356616907]
  We introduce the VIST-Character dataset, which provides rich character-centric annotations. Based on this dataset, we propose two new tasks: important character detection and character grounding in visual stories. We develop simple, unsupervised models based on distributional similarity and pre-trained vision-and-language models.
  arXiv  Detail & Related papers  (2023-03-30T18:24:06Z)
• Make-A-Story: Visual Memory Conditioned Consistent Story Generation [57.691064030235985]
  We propose a novel autoregressive diffusion-based framework with a visual memory module that implicitly captures the actor and background context. Our method outperforms prior state-of-the-art in generating frames with high visual quality. Our experiments for story generation on the MUGEN, the PororoSV and the FlintstonesSV dataset show that our method not only outperforms prior state-of-the-art in generating frames with high visual quality, but also models appropriate correspondences between the characters and the background.
  arXiv  Detail & Related papers  (2022-11-23T21:38:51Z)
• Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization [81.26077816854449]
  We first explore the use of constituency parse trees for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images.
  arXiv  Detail & Related papers  (2021-10-21T00:16:02Z)
• Improving Generation and Evaluation of Visual Stories via Semantic Consistency [72.00815192668193]
  Given a series of natural language captions, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform synthesis text-to-image models on this task. We present a number of improvements to prior modeling approaches, including the addition of a dual learning framework.
  arXiv  Detail & Related papers  (2021-05-20T20:42:42Z)
• ORD: Object Relationship Discovery for Visual Dialogue Generation [60.471670447176656]
  We propose an object relationship discovery (ORD) framework to preserve the object interactions for visual dialogue generation. A hierarchical graph convolutional network (HierGCN) is proposed to retain the object nodes and neighbour relationships locally, and then refines the object-object connections globally. Experiments have proved that the proposed method can significantly improve the quality of dialogue by utilising the contextual information of visual relationships.
  arXiv  Detail & Related papers  (2020-06-15T12:25:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.