Bridging Online and Offline Handwriting via Differentiable Physical Rendering

• URL: http://arxiv.org/abs/2608.03198v1
• Date: Tue, 04 Aug 2026 06:39:11 GMT
• Title: Bridging Online and Offline Handwriting via Differentiable Physical Rendering
• Abstract summary: We propose a unified online-offline handwriting generation framework via differentiable brush rendering.<n>The proposed framework consists of four core modules: 1) a text-to-stroke generator that predicts the target stroke conditioned on the given text and style image, 2) a brush parameter observer that extracts brush model parameters from style references, 3) a differentiable brush that maps a stroke sequence into a handwritten image, and 4) a zero-shot image refiner that refines rendered images.
• Score: 32.06536846087488
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Realistic handwritten text generation plays an important role in numerous applications, such as font design, biometric authentication, and robotic calligraphy. Existing methods are typically divided into two independent paradigms: online approaches that estimate handwriting trajectories and offline approaches that synthesize realistic handwriting images. While online models capture structural and temporal dynamics, they often lack fine-grained textures, whereas offline models reproduce realistic appearance but discard stroke order. However, unifying online and offline models remains challenging due to (1) the lack of an explicit physical model linking stroke kinematics to pixel-level appearance and (2) the absence of paired trajectory-image datasets. Moreover, enabling end-to-end learning requires a differentiable rendering process across motion and appearance domains. To address these challenges, we propose a compact physical brush model that bridges stroke dynamics and visual appearance, together with a differentiable rendering module that converts stroke trajectories into stylized images. By integrating these components, we propose a unified online-offline handwriting generation framework via differentiable brush rendering. The proposed framework consists of four core modules: 1) a text-to-stroke generator that predicts the target stroke conditioned on the given text and style image, 2) a brush parameter observer that extracts brush model parameters from style references, 3) a differentiable brush renderer that maps a stroke sequence and physical brush parameters into a handwritten image, and 4) a zero-shot image refiner that refines rendered images via diffusion models. Extensive experiments and real-world robotic calligraphy demonstrations validate our approach, achieving both structural and visual fidelity.

Related papers

• Single-Line Drawing Generation via Semantics-Driven Optimization [51.26529682867565]
  We present the first semantics-driven method for automatically generating single-line drawings in vector format.<n>Our approach leverages score distillation sampling to optimize the parameters of a uniform rational B-spline (URBS) curve.<n>Because our method generates a vectorized curve, it directly supports downstream fabrication processes such as embroidery, laser engraving and wire bending.
  arXiv  Detail & Related papers  (2026-06-01T08:46:22Z)
• Semantic-Structural Alignment for Generative Pictorial Charts [59.45629259524998]
  We present a generative framework for the automated synthesis of pictorial charts.<n>We frame the problem as a dual-conditioned generation task guided by two parallel external control signals.<n>Our method produces pictorial charts that are both artistically compelling and structurally consistent.
  arXiv  Detail & Related papers  (2026-05-05T05:20:46Z)
• Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting [60.15416769662556]
  We study the problem of learning human-like brushstroke generation from a small set of hand-drawn samples.<n>We propose StrokeDiff, a diffusion-based framework with Smooth Regularization (SmR)<n>We show how the learned primitives can be made controllable through a Bézier-based conditioning module.
  arXiv  Detail & Related papers  (2026-03-01T13:42:35Z)
• VideoSketcher: Video Models Prior Enable Versatile Sequential Sketch Generation [73.23035143627598]
  Most generative models treat sketches as static images, overlooking the temporal structure that underlies creative drawing.<n>We present a data-efficient approach for sequential sketch generation that adapts pretrained text-to-video diffusion models.<n>Our method generates high-quality sketches that closely follow text-specified orderings while exhibiting rich visual detail.
  arXiv  Detail & Related papers  (2026-02-17T18:55:03Z)
• ScriptViT: Vision Transformer-Based Personalized Handwriting Generation [0.0]
  styled handwriting generation aims to synthesize handwritten text that looks both realistic and aligned with a specific writer's style.<n>We introduce a Vision Transformer-based style encoder that learns global stylistic patterns from multiple reference images.<n>We then integrate these style cues with the target text using a cross-attention mechanism, enabling the system to produce handwritten images that more faithfully reflect the intended style.
  arXiv  Detail & Related papers  (2025-11-23T06:38:23Z)
• Birth of a Painting: Differentiable Brushstroke Reconstruction [25.61763988336406]
  Painting embodies a unique form of visual storytelling, where the creation process is as significant as the final artwork.<n>Our approach produces realistic and stylized appearances, offering a unified model for digital painting.
  arXiv  Detail & Related papers  (2025-11-17T09:55:53Z)
• Every Painting Awakened: A Training-free Framework for Painting-to-Animation Generation [25.834500552609136]
  We introduce a training-free framework specifically designed to bring real-world static paintings to life through image-to-video (I2V) synthesis.<n>Existing I2V methods, primarily trained on natural video datasets, often struggle to generate dynamic outputs from static paintings.<n>Our framework enables plug-and-play integration with existing I2V methods, making it an ideal solution for animating real-world paintings.
  arXiv  Detail & Related papers  (2025-03-31T05:25:49Z)
• AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation [61.77946020543875]
  We propose a framework for translating raw descriptions with complex semantics into semantically corresponding images. Our framework consists of two components: a projection module from Text Embeddings to Image Embeddings based on prompts, and an adapted image generation module built on StyleGAN. Benefiting from the pre-trained models, our method can handle complex descriptions and does not require external paired data for training.
  arXiv  Detail & Related papers  (2022-09-07T13:53:54Z)
• Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model [31.652827838300915]
  We propose a unified framework supporting a three-dimensional control over the image synthesis from sketches and strokes based on diffusion models. Our framework achieves state-of-the-art performance while providing flexibility in generating customized images with control over shape, color, and realism. Our method unleashes applications such as editing on real images, generation with partial sketches and strokes, and multi-domain multi-modal synthesis.
  arXiv  Detail & Related papers  (2022-08-26T13:59:26Z)
• Improving Generation and Evaluation of Visual Stories via Semantic Consistency [72.00815192668193]
  Given a series of natural language captions, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform synthesis text-to-image models on this task. We present a number of improvements to prior modeling approaches, including the addition of a dual learning framework.
  arXiv  Detail & Related papers  (2021-05-20T20:42:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.