Doc To The Future: Infomorphs for Interactive, Multimodal Document Transformation and Generation

• URL: http://arxiv.org/abs/2602.23366v1
• Date: Sun, 14 Dec 2025 16:25:19 GMT
• Title: Doc To The Future: Infomorphs for Interactive, Multimodal Document Transformation and Generation
• Abstract summary: We introduce the notion of "infomorphs" which are modular, user-steerable, AI-augmented transformations that support controlled synthesis, and restructuring of information across formats and modalities.<n>We propose a design space that leverage infomorph-driven document creation by combining Generative AI techniques with user intent and desired information context.<n>DocuCraft allows users to chain together infomorphs that perform operations such as page extraction, content summarization, reformatting, and generation, leveraging Generative AI at each stage to support rich, cross-document and cross-modal transformations.
• Score: 4.665470854146965
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Creating new documents by synthesizing information from existing sources is an important part of knowledge work in many domains. This process often involves gathering content from multiple documents, organizing it, and then transforming it into new forms such as reports, slides, or spreadsheets. While recent advances in Generative AI have shown potential in automating parts of this process, they often provide limited user control over the handling of multimodal inputs and outputs. In this work, we introduce the notion of "infomorphs" which are modular, user-steerable, AI-augmented transformations that support controlled synthesis, and restructuring of information across formats and modalities. We propose a design space that leverage infomorph-driven workflows to enable flexible, interactive, and multimodal document creation by combining Generative AI techniques with user intent and desired information context. As a concrete instantiation of this design space, we present DocuCraft, a canvas-based interface to visually compose infomorph workflows. DocuCraft allows users to chain together infomorphs that perform operations such as page extraction, content summarization, reformatting, and generation, leveraging Generative AI at each stage to support rich, cross-document and cross-modal transformations. We demonstrate the capabilities of DocuCraft through an example-driven usage scenario that spans across different facets of common knowledge work tasks illustrating its support for fluid, human-in-the-loop document synthesis and highlights opportunities for more transparent and modular interaction for Generative AI-assisted information work.

Related papers

• I-WebGenBench : Evaluating Interactivity in LLM-Generated Scientific Web Applications [33.31056974240273]
  We propose a Paper-to-Interactive-System Agent that converts research papers into executable interactive web systems.<n>Given a PDF paper, the agent performs end-to-end processing without human intervention.<n>Experiments show that PaperVoyager significantly improves the quality of generated interactive systems.
  arXiv  Detail & Related papers  (2026-05-30T14:34:02Z)
• Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training [68.94182767962914]
  We introduce a framework consisting of a planner and a visualizer.<n>The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly.<n>These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency.
  arXiv  Detail & Related papers  (2026-03-26T17:50:37Z)
• PaperVoyager : Building Interactive Web with Visual Language Models [35.73710997735284]
  We propose a Paper-to-Interactive-System Agent that converts research papers into executable interactive web systems.<n>Given a PDF paper, the agent performs end-to-end processing without human intervention.<n>Experiments show that PaperVoyager significantly improves the quality of generated interactive systems.
  arXiv  Detail & Related papers  (2026-03-24T09:42:27Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• Bridging Collaborative Filtering and Large Language Models with Dynamic Alignment, Multimodal Fusion and Evidence-grounded Explanations [1.3702600718499687]
  We develop an online adaptation mechanism that incorporates new user interactions through lightweight modules.<n>We create a unified representation that seamlessly combines collaborative signals with visual and audio features.<n>Our approach maintains the efficiency of frozen base models while adding minimal computational overhead, making it practical for real-world deployment.
  arXiv  Detail & Related papers  (2025-10-02T02:43:24Z)
• Textual-to-Visual Iterative Self-Verification for Slide Generation [46.99825956909532]
  We decompose the task of generating missing presentation slides into two key components: content generation and layout generation.<n>Our approach significantly outperforms baseline methods in terms of alignment, logical flow, visual appeal, and readability.
  arXiv  Detail & Related papers  (2025-02-21T12:21:09Z)
• ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation [87.39861573270173]
  We introduce the novel task of prompt-adaptive workflow generation, where the goal is to automatically tailor a workflow to each user prompt. We propose two LLM-based approaches to tackle this task: a tuning-based method that learns from user-preference data, and a training-free method that uses the LLM to select existing flows. Our work shows that prompt-dependent flow prediction offers a new pathway to improving text-to-image generation quality, complementing existing research directions in the field.
  arXiv  Detail & Related papers  (2024-10-02T16:43:24Z)
• AppAgent v2: Advanced Agent for Flexible Mobile Interactions [57.98933460388985]
  This work introduces a novel LLM-based multimodal agent framework for mobile devices.<n>Our agent constructs a flexible action space that enhances adaptability across various applications.<n>Our results demonstrate the framework's superior performance, confirming its effectiveness in real-world scenarios.
  arXiv  Detail & Related papers  (2024-08-05T06:31:39Z)
• Hierarchical Multi-modal Transformer for Cross-modal Long Document Classification [74.45521856327001]
  How to classify long documents with hierarchical structure texts and embedding images is a new problem. We propose a novel approach called Hierarchical Multi-modal Transformer (HMT) for cross-modal long document classification. Our approach uses a multi-modal transformer and a dynamic multi-scale multi-modal transformer to model the complex relationships between image features, and the section and sentence features.
  arXiv  Detail & Related papers  (2024-07-14T07:12:25Z)
• DocSynthv2: A Practical Autoregressive Modeling for Document Generation [43.84027661517748]
  This paper proposes a novel approach called Doc Synthv2 through the development of a simple yet effective autoregressive structured model. Our model, distinct in its integration of both layout and textual cues, marks a step beyond existing layout-generation approaches.
  arXiv  Detail & Related papers  (2024-06-12T16:00:16Z)
• An Interactive Agent Foundation Model [49.77861810045509]
  We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction. We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
  arXiv  Detail & Related papers  (2024-02-08T18:58:02Z)
• Towards Flexible Multi-modal Document Models [27.955214767628107]
  In this work, we attempt at building a holistic model that can jointly solve many different design tasks. Our model, which we denote by FlexDM, treats vector graphic documents as a harmonious set of multi-modal elements. Experimental results corroborate that our single FlexDM is able to successfully solve a multitude of different design tasks.
  arXiv  Detail & Related papers  (2023-03-31T17:59:56Z)
• DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents [76.19748112897177]
  We present a novel task and approach for document-to-slide generation. We propose a hierarchical sequence-to-sequence approach to tackle our task in an end-to-end manner. Our approach exploits the inherent structures within documents and slides and incorporates paraphrasing and layout prediction modules to generate slides.
  arXiv  Detail & Related papers  (2021-01-28T03:21:17Z)
• Augmenting Transformers with KNN-Based Composite Memory for Dialogue [22.677844670431885]
  We propose augmenting generative Transformer neural networks with KNN-based Information Fetching (KIF) modules. We apply these modules to generative dialog modeling, a challenging task where information must be flexibly retrieved and incorporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge required for knowledgeable but engaging dialog from Wikipedia, images, and human-written dialog utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.
  arXiv  Detail & Related papers  (2020-04-27T12:37:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.