Abstract: In this paper, we propose a fully automatic system for generating comic books
from videos without any human intervention. Given an input video along with its
subtitles, our approach first extracts informative keyframes by analyzing the
subtitles, and stylizes keyframes into comic-style images. Then, we propose a
novel automatic multi-page layout framework, which can allocate the images
across multiple pages and synthesize visually interesting layouts based on the
rich semantics of the images (e.g., importance and inter-image relation).
Finally, as opposed to using the same type of balloon as in previous works, we
propose an emotion-aware balloon generation method to create different types of
word balloons by analyzing the emotion of subtitles and audios. Our method is
able to vary balloon shapes and word sizes in balloons in response to different
emotions, leading to more enriched reading experience. Once the balloons are
generated, they are placed adjacent to their corresponding speakers via speaker
detection. Our results show that our method, without requiring any user inputs,
can generate high-quality comic pages with visually rich layouts and balloons.
Our user studies also demonstrate that users prefer our generated results over
those by state-of-the-art comic generation systems.