MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-4

• URL: http://arxiv.org/abs/2406.00971v1
• Date: Mon, 3 Jun 2024 03:59:29 GMT
• Title: MiniGPT-Reverse-Designing: Predicting Image Adjustments Utilizing MiniGPT-4
• Authors: Vahid Azizi, Fatemeh Koochaki,
• Abstract summary: Vision-Language Models (VLMs) have recently seen significant advancements through integrating with Large Language Models (LLMs) In this paper, we extend and fine-tune MiniGPT-4 for the reverse designing task.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) have recently seen significant advancements through integrating with Large Language Models (LLMs). The VLMs, which process image and text modalities simultaneously, have demonstrated the ability to learn and understand the interaction between images and texts across various multi-modal tasks. Reverse designing, which could be defined as a complex vision-language task, aims to predict the edits and their parameters, given a source image, an edited version, and an optional high-level textual edit description. This task requires VLMs to comprehend the interplay between the source image, the edited version, and the optional textual context simultaneously, going beyond traditional vision-language tasks. In this paper, we extend and fine-tune MiniGPT-4 for the reverse designing task. Our experiments demonstrate the extensibility of off-the-shelf VLMs, specifically MiniGPT-4, for more complex tasks such as reverse designing. Code is available at this \href{https://github.com/VahidAz/MiniGPT-Reverse-Designing}

Related papers

• Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation [31.080061818510003]
  Document Image Machine Translation (DIMT) aims to translate text within document images.<n>We introduce M4Doc, a novel single-to-mix modality alignment framework leveraging Multimodal Large Language Models (MLLMs)<n>M4Doc aligns an image-only encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets.
  arXiv  Detail & Related papers  (2025-07-10T09:18:06Z)
• MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval [50.062817677022586]
  Zero-Shot Image Retrieval (ZS-CIR) methods typically train adapters that convert reference images into pseudo-text tokens.<n>We propose MLLM-Guided VLM Fine-Tuning with Joint Inference (MVFT-JI) to construct two complementary training tasks using only unlabeled images.
  arXiv  Detail & Related papers  (2025-05-26T08:56:59Z)
• Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks [62.758680527838436]
  Leopard is a vision-language model for handling vision-language tasks involving multiple text-rich images. First, we curated about one million high-quality multimodal instruction-tuning data, tailored to text-rich, multi-image scenarios. Second, we developed an adaptive high-resolution multi-image encoding module to dynamically optimize the allocation of visual sequence length.
  arXiv  Detail & Related papers  (2024-10-02T16:55:01Z)
• VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models [76.94378391979228]
  We introduce a new, more demanding task known as Interleaved Image-Text (IITC) This task challenges models to discern and disregard superfluous elements in both images and text to accurately answer questions. In support of this task, we further craft a new VEGA dataset, tailored for the IITC task on scientific content, and devised a subtask, Image-Text Association (ITA)
  arXiv  Detail & Related papers  (2024-06-14T17:59:40Z)
• VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text [80.24176572093512]
  We introduce Visual Caption Restoration (VCR), a vision-language task that challenges models to accurately restore partially obscured texts using pixel-level hints within images. This task stems from the observation that text embedded in images is intrinsically different from common visual elements and natural language due to the need to align the modalities of vision, text, and text embedded in images.
  arXiv  Detail & Related papers  (2024-06-10T16:58:48Z)
• FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction [66.98008357232428]
  We propose FineMatch, a new aspect-based fine-grained text and image matching benchmark. FineMatch focuses on text and image mismatch detection and correction. We show that models trained on FineMatch demonstrate enhanced proficiency in detecting fine-grained text and image mismatches.
  arXiv  Detail & Related papers  (2024-04-23T03:42:14Z)
• Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models [22.545127591893028]
  Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA) This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. We present a novel approach, Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual information to enhance the capability of MLLMs in VQA.
  arXiv  Detail & Related papers  (2024-04-06T05:59:02Z)
• Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want [58.091825321168514]
  We introduce the Draw-and-Understand project: a new model, a multi-domain dataset, and a challenging benchmark for visual prompting. Specifically, we propose a new end-to-end trained Multimodal Large Language Model (MLLM) that connects a vision encoder, a visual prompt encoder and an LLM. To advance visual prompting research for MLLMs, we introduce MDVP-Data and MDVP-Bench.
  arXiv  Detail & Related papers  (2024-03-29T16:26:20Z)
• Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models [18.772045053892885]
  State-of-the-art Large Multi-Modal Models (LMMs) have demonstrated exceptional capabilities in vision-language tasks. Existing prompting techniques for LMMs focus on either improving textual reasoning or leveraging tools for image preprocessing. We propose Scaffold prompting that scaffolds coordinates to promote vision-language coordination.
  arXiv  Detail & Related papers  (2024-02-19T11:23:53Z)
• Vision-by-Language for Training-Free Compositional Image Retrieval [78.60509831598745]
  Compositional Image Retrieval (CIR) aims to retrieve the relevant target image in a database. Recent research sidesteps this need by using large-scale vision-language models (VLMs) We propose to tackle CIR in a training-free manner via Vision-by-Language (CIReVL)
  arXiv  Detail & Related papers  (2023-10-13T17:59:38Z)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists [66.85125112199898]
  We develop a unified language interface for computer vision tasks that abstracts away task-specific design choices. Our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models.
  arXiv  Detail & Related papers  (2023-09-30T14:26:43Z)
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models [41.84885546518666]
  GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text. We present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced large language model. We also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images.
  arXiv  Detail & Related papers  (2023-04-20T18:25:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.