Boosting Diffusion-Based Text Image Super-Resolution Model Towards Generalized Real-World Scenarios

• URL: http://arxiv.org/abs/2503.07232v2
• Date: Tue, 11 Mar 2025 06:00:49 GMT
• Title: Boosting Diffusion-Based Text Image Super-Resolution Model Towards Generalized Real-World Scenarios
• Authors: Chenglu Pan, Xiaogang Xu, Ganggui Ding, Yunke Zhang, Wenbo Li, Jiarong Xu, Qingbiao Wu,
• Abstract summary: We introduce a novel framework aimed at improving the generalization ability of diffusion models for text image super-resolution (SR)<n>We propose a progressive data sampling strategy that incorporates diverse image types at different stages of training, stabilizing the convergence and improving the generalization.<n>Experiments on real-world datasets demonstrate that our approach not only produces text images with more realistic visual appearances but also improves the accuracy of text structure.
• Score: 30.800865323585377
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Restoring low-resolution text images presents a significant challenge, as it requires maintaining both the fidelity and stylistic realism of the text in restored images. Existing text image restoration methods often fall short in hard situations, as the traditional super-resolution models cannot guarantee clarity, while diffusion-based methods fail to maintain fidelity. In this paper, we introduce a novel framework aimed at improving the generalization ability of diffusion models for text image super-resolution (SR), especially promoting fidelity. First, we propose a progressive data sampling strategy that incorporates diverse image types at different stages of training, stabilizing the convergence and improving the generalization. For the network architecture, we leverage a pre-trained SR prior to provide robust spatial reasoning capabilities, enhancing the model's ability to preserve textual information. Additionally, we employ a cross-attention mechanism to better integrate textual priors. To further reduce errors in textual priors, we utilize confidence scores to dynamically adjust the importance of textual features during training. Extensive experiments on real-world datasets demonstrate that our approach not only produces text images with more realistic visual appearances but also improves the accuracy of text structure.

Related papers

• Contrastive Learning Guided Latent Diffusion Model for Image-to-Image Translation [7.218556478126324]
  diffusion model has demonstrated superior performance in diverse and high-quality images for text-guided image translation. We propose pix2pix-zeroCon, a zero-shot diffusion-based method that eliminates the need for additional training by leveraging patch-wise contrastive loss. Our approach requires no additional training and operates directly on a pre-trained text-to-image diffusion model.
  arXiv  Detail & Related papers  (2025-03-26T12:15:25Z)
• Beyond Pixels: Text Enhances Generalization in Real-World Image Restoration [47.942948541067544]
  We propose using text as an auxiliary invariant representation to reactivate the generative capabilities of diffusion-based restoration models.<n>We introduce Res-Captioner, a module that generates enhanced textual descriptions tailored to image content and degradation levels.<n>We present RealIR, a new benchmark designed to capture diverse real-world scenarios.
  arXiv  Detail & Related papers  (2024-12-01T16:36:22Z)
• Coherent and Multi-modality Image Inpainting via Latent Space Optimization [61.99406669027195]
  PILOT (intextbfPainting vtextbfIa textbfLatent textbfOptextbfTimization) is an optimization approach grounded on a novel textitsemantic centralization and textitbackground preservation loss. Our method searches latent spaces capable of generating inpainted regions that exhibit high fidelity to user-provided prompts while maintaining coherence with the background.
  arXiv  Detail & Related papers  (2024-07-10T19:58:04Z)
• Efficient Personalized Text-to-image Generation by Leveraging Textual Subspace [52.24866347353916]
  We propose an efficient method to explore the target embedding in a textual subspace. We also propose an efficient selection strategy for determining the basis of the textual subspace. Our method opens the door to more efficient representation learning for personalized text-to-image generation.
  arXiv  Detail & Related papers  (2024-06-30T06:41:21Z)
• DaLPSR: Leverage Degradation-Aligned Language Prompt for Real-World Image Super-Resolution [19.33582308829547]
  This paper proposes to leverage degradation-aligned language prompt for accurate, fine-grained, and high-fidelity image restoration. The proposed method achieves a new state-of-the-art perceptual quality level.
  arXiv  Detail & Related papers  (2024-06-24T09:30:36Z)
• ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models [52.23899502520261]
  We introduce a novel framework named, ARTIST, which incorporates a dedicated textual diffusion model to focus on the learning of text structures specifically.<n>We finetune a visual diffusion model, enabling it to assimilate textual structure information from the pretrained textual model.<n>This disentangled architecture design and training strategy significantly enhance the text rendering ability of the diffusion models for text-rich image generation.
  arXiv  Detail & Related papers  (2024-06-17T19:31:24Z)
• Diffusion-based Blind Text Image Super-Resolution [20.91578221617732]
  We propose an Image Diffusion Model (IDM) to restore text images with realistic styles. For diffusion models, they are not only suitable for modeling realistic image distribution but also appropriate for learning text distribution. We also propose a Text Diffusion Model (TDM) for text recognition which can guide IDM to generate text images with correct structures.
  arXiv  Detail & Related papers  (2023-12-13T06:03:17Z)
• CoSeR: Bridging Image and Language for Cognitive Super-Resolution [74.24752388179992]
  We introduce the Cognitive Super-Resolution (CoSeR) framework, empowering SR models with the capacity to comprehend low-resolution images. We achieve this by marrying image appearance and language understanding to generate a cognitive embedding. To further improve image fidelity, we propose a novel condition injection scheme called "All-in-Attention"
  arXiv  Detail & Related papers  (2023-11-27T16:33:29Z)
• DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis [80.54273334640285]
  We propose a novel one-stage text-to-image backbone that directly synthesizes high-resolution images without entanglements between different generators. We also propose a novel Target-Aware Discriminator composed of Matching-Aware Gradient Penalty and One-Way Output. Compared with current state-of-the-art methods, our proposed DF-GAN is simpler but more efficient to synthesize realistic and text-matching images.
  arXiv  Detail & Related papers  (2020-08-13T12:51:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.