DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks

• URL: http://arxiv.org/abs/2405.04408v1
• Date: Tue, 7 May 2024 15:35:43 GMT
• Title: DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks
• Authors: Jiaxin Zhang, Dezhi Peng, Chongyu Liu, Peirong Zhang, Lianwen Jin,
• Abstract summary: We propose DocRes, a model that unifies five document image restoration tasks including dewarping, deshadowing, appearance enhancement, deblurring, and binarization. To instruct DocRes to perform different restoration tasks, we propose a novel visual prompt approach called Dynamic Task-Specific Prompt (DTSPrompt) DTSPrompt is more flexible than prior visual prompt approaches as it can be seamlessly applied and adapted to inputs with high and variable resolutions.
• Score: 38.6455393290578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Document image restoration is a crucial aspect of Document AI systems, as the quality of document images significantly influences the overall performance. Prevailing methods address distinct restoration tasks independently, leading to intricate systems and the incapability to harness the potential synergies of multi-task learning. To overcome this challenge, we propose DocRes, a generalist model that unifies five document image restoration tasks including dewarping, deshadowing, appearance enhancement, deblurring, and binarization. To instruct DocRes to perform various restoration tasks, we propose a novel visual prompt approach called Dynamic Task-Specific Prompt (DTSPrompt). The DTSPrompt for different tasks comprises distinct prior features, which are additional characteristics extracted from the input image. Beyond its role as a cue for task-specific execution, DTSPrompt can also serve as supplementary information to enhance the model's performance. Moreover, DTSPrompt is more flexible than prior visual prompt approaches as it can be seamlessly applied and adapted to inputs with high and variable resolutions. Experimental results demonstrate that DocRes achieves competitive or superior performance compared to existing state-of-the-art task-specific models. This underscores the potential of DocRes across a broader spectrum of document image restoration tasks. The source code is publicly available at https://github.com/ZZZHANG-jx/DocRes

Related papers

• Learning A Low-Level Vision Generalist via Visual Task Prompt [43.54563263106761]
  We propose a Visual task Prompt-based Image Processing (VPIP) framework to overcome these challenges. VPIP employs visual task prompts to manage tasks with different input-target domains and allows flexible selection of backbone network. Based on the VPIP framework, we train a low-level vision generalist model, namely GenLV, on 30 diverse tasks.
  arXiv  Detail & Related papers  (2024-08-16T08:37:56Z)
• PromptFix: You Prompt and We Fix the Photo [84.69812824355269]
  Diffusion models equipped with language models demonstrate excellent controllability in image generation tasks. The lack of diverse instruction-following data hampers the development of models. We propose PromptFix, a framework that enables diffusion models to follow human instructions.
  arXiv  Detail & Related papers  (2024-05-27T03:13:28Z)
• Unified-Width Adaptive Dynamic Network for All-In-One Image Restoration [50.81374327480445]
  We introduce a novel concept positing that intricate image degradation can be represented in terms of elementary degradation. We propose the Unified-Width Adaptive Dynamic Network (U-WADN), consisting of two pivotal components: a Width Adaptive Backbone (WAB) and a Width Selector (WS) The proposed U-WADN achieves better performance while simultaneously reducing up to 32.3% of FLOPs and providing approximately 15.7% real-time acceleration.
  arXiv  Detail & Related papers  (2024-01-24T04:25:12Z)
• SPIRE: Semantic Prompt-Driven Image Restoration [66.26165625929747]
  We develop SPIRE, a Semantic and restoration Prompt-driven Image Restoration framework. Our approach is the first framework that supports fine-level instruction through language-based quantitative specification of the restoration strength. Our experiments demonstrate the superior restoration performance of SPIRE compared to the state of the arts.
  arXiv  Detail & Related papers  (2023-12-18T17:02:30Z)
• DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding [91.17151775296234]
  This work presents DocPedia, a novel large multimodal model (LMM) for versatile OCR-free document understanding. Unlike existing work either struggle with high-resolution documents or give up the large language model thus vision or language ability constrained, our DocPedia directly processes visual input in the frequency domain rather than the pixel space.
  arXiv  Detail & Related papers  (2023-11-20T14:42:25Z)
• Unifying Image Processing as Visual Prompting Question Answering [62.84955983910612]
  Image processing is a fundamental task in computer vision, which aims at enhancing image quality and extracting essential features for subsequent vision applications. Traditionally, task-specific models are developed for individual tasks and designing such models requires distinct expertise. We propose a universal model for general image processing that covers image restoration, image enhancement, image feature extraction tasks.
  arXiv  Detail & Related papers  (2023-10-16T15:32:57Z)
• ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration [46.87227160492818]
  We present Degradation-aware Visual Prompts, which encode various types of image degradation into unified visual prompts. These degradation-aware prompts provide control over image processing and allow weighted combinations for customized image restoration. We then leverage degradation-aware visual prompts to establish a controllable universal model for image restoration.
  arXiv  Detail & Related papers  (2023-06-23T17:59:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.