InkSight: Offline-to-Online Handwriting Conversion by Learning to Read and Write

• URL: http://arxiv.org/abs/2402.05804v3
• Date: Sun, 08 Dec 2024 21:05:43 GMT
• Title: InkSight: Offline-to-Online Handwriting Conversion by Learning to Read and Write
• Authors: Blagoj Mitrevski, Arina Rak, Julian Schnitzler, Chengkun Li, Andrii Maksai, Jesse Berent, Claudiu Musat,
• Abstract summary: InkSight aims to empower physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting) Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image.
• Score: 7.4539464693425925
• License:
• Abstract: Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.

Related papers

• MetaScript: Few-Shot Handwritten Chinese Content Generation via Generative Adversarial Networks [15.037121719502606]
  We propose MetaScript, a novel content generation system designed to address the diminishing presence of personal handwriting styles in the digital representation of Chinese characters. Our approach harnesses the power of few-shot learning to generate Chinese characters that retain the individual's unique handwriting style and maintain the efficiency of digital typing.
  arXiv  Detail & Related papers  (2023-12-25T17:31:19Z)
• Sampling and Ranking for Digital Ink Generation on a tight computational budget [69.15275423815461]
  We study ways to maximize the quality of the output of a trained digital ink generative model. We use and compare the effect of multiple sampling and ranking techniques, in the first ablation study of its kind in the digital ink domain.
  arXiv  Detail & Related papers  (2023-06-02T09:55:15Z)
• ALADIN-NST: Self-supervised disentangled representation learning of artistic style through Neural Style Transfer [60.6863849241972]
  We learn a representation of visual artistic style more strongly disentangled from the semantic content depicted in an image. We show that strongly addressing the disentanglement of style and content leads to large gains in style-specific metrics.
  arXiv  Detail & Related papers  (2023-04-12T10:33:18Z)
• Character-Aware Models Improve Visual Text Rendering [57.19915686282047]
  Current image generation models struggle to reliably produce well-formed visual text. Character-aware models provide large gains on a novel spelling task. Our models set a much higher state-of-the-art on visual spelling, with 30+ point accuracy gains over competitors on rare words.
  arXiv  Detail & Related papers  (2022-12-20T18:59:23Z)
• PART: Pre-trained Authorship Representation Transformer [64.78260098263489]
  Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage. Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors. We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
  arXiv  Detail & Related papers  (2022-09-30T11:08:39Z)
• I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches [74.63313641583602]
  We propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects. Our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications.
  arXiv  Detail & Related papers  (2022-05-09T04:23:36Z)
• Content and Style Aware Generation of Text-line Images for Handwriting Recognition [4.301658883577544]
  We propose a generative method for handwritten text-line images conditioned on both visual appearance and textual content. Our method is able to produce long text-line samples with diverse handwriting styles.
  arXiv  Detail & Related papers  (2022-04-12T05:52:03Z)
• Letter-level Online Writer Identification [86.13203975836556]
  We focus on a novel problem, letter-level online writer-id, which requires only a few trajectories of written letters as identification cues. A main challenge is that a person often writes a letter in different styles from time to time. We refer to this problem as the variance of online writing styles (Var-O-Styles)
  arXiv  Detail & Related papers  (2021-12-06T07:21:53Z)
• Spectral Graph-based Features for Recognition of Handwritten Characters: A Case Study on Handwritten Devanagari Numerals [0.0]
  We propose an approach that exploits the robust graph representation and spectral graph embedding concept to represent handwritten characters. For corroboration of the efficacy of the proposed method, extensive experiments were carried out on the standard handwritten numeral Computer Vision Pattern Recognition, Unit of Indian Statistical Institute Kolkata dataset.
  arXiv  Detail & Related papers  (2020-07-07T08:40:08Z)
• FCN+RL: A Fully Convolutional Network followed by Refinement Layers to Offline Handwritten Signature Segmentation [3.3144312096837325]
  We propose an approach to locate and extract the pixels of handwritten signatures on identification documents. The technique is based on a fully convolutional encoder-decoder network combined with a block of refinement layers for the alpha channel of the predicted image.
  arXiv  Detail & Related papers  (2020-05-28T18:47:10Z)
• Spatio-Temporal Handwriting Imitation [11.54523121769666]
  Subdividing the process into smaller subtasks makes it possible to imitate someone's handwriting with a high chance to be visually indistinguishable for humans. We show that also a typical writer identification system can partially be fooled by the created fake handwritings.
  arXiv  Detail & Related papers  (2020-03-24T00:46:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.