Adapting Large VLMs with Iterative and Manual Instructions for Generative Low-light Enhancement

• URL: http://arxiv.org/abs/2507.18064v1
• Date: Thu, 24 Jul 2025 03:35:20 GMT
• Title: Adapting Large VLMs with Iterative and Manual Instructions for Generative Low-light Enhancement
• Authors: Xiaoran Sun, Liyan Wang, Cong Wang, Yeying Jin, Kin-man Lam, Zhixun Su, Yang Yang, Jinshan Pan,
• Abstract summary: Most low-light image enhancement methods rely on pre-trained model priors, low-light inputs, or both.<n>We propose VLM-IMI, a novel framework that leverages large vision-language models with iterative and manual instructions.<n>VLM-IMI incorporates textual descriptions of the desired normal-light content as enhancement cues, enabling semantically informed restoration.
• Score: 41.66776033752888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most existing low-light image enhancement (LLIE) methods rely on pre-trained model priors, low-light inputs, or both, while neglecting the semantic guidance available from normal-light images. This limitation hinders their effectiveness in complex lighting conditions. In this paper, we propose VLM-IMI, a novel framework that leverages large vision-language models (VLMs) with iterative and manual instructions (IMIs) for LLIE. VLM-IMI incorporates textual descriptions of the desired normal-light content as enhancement cues, enabling semantically informed restoration. To effectively integrate cross-modal priors, we introduce an instruction prior fusion module, which dynamically aligns and fuses image and text features, promoting the generation of detailed and semantically coherent outputs. During inference, we adopt an iterative and manual instruction strategy to refine textual instructions, progressively improving visual quality. This refinement enhances structural fidelity, semantic alignment, and the recovery of fine details under extremely low-light conditions. Extensive experiments across diverse scenarios demonstrate that VLM-IMI outperforms state-of-the-art methods in both quantitative metrics and perceptual quality. The source code is available at https://github.com/sunxiaoran01/VLM-IMI.

Related papers

• SAIGFormer: A Spatially-Adaptive Illumination-Guided Network for Low-Light Image Enhancement [58.79901582809091]
  Recent Transformer-based low-light enhancement methods have made promising progress in recovering global illumination.<n>Recent Transformer-based low-light enhancement methods have made promising progress in recovering global illumination.<n>We present a Spatially-Adaptive Illumination-Guided Transformer framework that enables accurate illumination restoration.
  arXiv  Detail & Related papers  (2025-07-21T11:38:56Z)
• Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models [56.84206059390887]
  We propose textbfLightD, a novel framework that generates natural adversarial samples for vision-and-language pretraining models.<n>LightD expands the optimization space while ensuring perturbations align with scene semantics.
  arXiv  Detail & Related papers  (2025-05-30T05:30:02Z)
• Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning [10.761218096540976]
  Multimodal reasoning in Large Language Models (LLMs) struggles with incomplete knowledge and hallucination artifacts.<n>We propose Vision-align-to-Language integrated Knowledge Graph (VaLiK), a novel approach for constructing Multimodal Knowledge Graphs.
  arXiv  Detail & Related papers  (2025-03-17T09:31:14Z)
• TSCnet: A Text-driven Semantic-level Controllable Framework for Customized Low-Light Image Enhancement [30.498816319802412]
  We propose a new light enhancement task and a new framework that provides customized lighting control through prompt-driven, semantic-level, and quantitative brightness adjustments.<n> Experimental results on benchmark datasets demonstrate our framework's superior performance at increasing visibility, maintaining natural color balance, and amplifying fine details without creating artifacts.
  arXiv  Detail & Related papers  (2025-03-11T08:30:50Z)
• From Visuals to Vocabulary: Establishing Equivalence Between Image and Text Token Through Autoregressive Pre-training in MLLMs [23.011836329934255]
  Vision Dynamic Embedding-Guided Pretraining (VDEP) is a hybrid autoregressive training paradigm for MLLMs.<n>The proposed method seamlessly integrates into standard models without architectural changes.<n> Experiments on 13 benchmarks show VDEP outperforms baselines, surpassing existing methods.
  arXiv  Detail & Related papers  (2025-02-13T09:04:28Z)
• Natural Language Supervision for Low-light Image Enhancement [0.0]
  We introduce a Natural Language Supervision (NLS) strategy, which learns feature maps from text corresponding to images.<n>We also design a Textual Guidance Conditioning Mechanism (TCM) that incorporates the connections between image regions and sentence words.<n>In order to effectively identify and merge features from various levels of image and textual information, we design an Information Fusion Attention (IFA) module.
  arXiv  Detail & Related papers  (2025-01-11T13:53:10Z)
• EMMA: Efficient Visual Alignment in Multi-Modal LLMs [56.03417732498859]
  EMMA is a lightweight cross-modality module designed to efficiently fuse visual and textual encodings.<n>EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations.
  arXiv  Detail & Related papers  (2024-10-02T23:00:31Z)
• GLARE: Low Light Image Enhancement via Generative Latent Feature based Codebook Retrieval [80.96706764868898]
  We present a new Low-light Image Enhancement (LLIE) network via Generative LAtent feature based codebook REtrieval (GLARE) We develop a generative Invertible Latent Normalizing Flow (I-LNF) module to align the LL feature distribution to NL latent representations, guaranteeing the correct code retrieval in the codebook. Experiments confirm the superior performance of GLARE on various benchmark datasets and real-world data.
  arXiv  Detail & Related papers  (2024-07-17T09:40:15Z)
• Progressive Multi-modal Conditional Prompt Tuning [92.50645776024624]
  Pre-trained vision-language models (VLMs) have shown remarkable generalization capabilities via prompting. We propose a novel method, Progressive Multi-modal conditional Prompt Tuning (ProMPT) ProMPT exploits a recurrent structure, optimizing and aligning V-L features by iteratively utilizing image and current encoding information.
  arXiv  Detail & Related papers  (2024-04-18T02:40:31Z)
• CodeEnhance: A Codebook-Driven Approach for Low-Light Image Enhancement [97.95330185793358]
  Low-light image enhancement (LLIE) aims to improve low-illumination images.<n>Existing methods face two challenges: uncertainty in restoration from diverse brightness degradations and loss of texture and color information.<n>We propose a novel enhancement approach, CodeEnhance, by leveraging quantized priors and image refinement.
  arXiv  Detail & Related papers  (2024-04-08T07:34:39Z)
• Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance [51.30560006045442]
  Image-gRounded guIdaNcE (MARINE) is a framework that is both training-free and API-free.<n>MARINE effectively and efficiently reduces object hallucinations during inference by introducing image-grounded guidance to LVLMs.<n>Our framework's flexibility further allows for the integration of multiple vision models, enabling more reliable and robust object-level guidance.
  arXiv  Detail & Related papers  (2024-02-13T18:59:05Z)
• Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment [15.180715595425864]
  We introduce a novel method to improve the prompt learning of vision-language models by incorporating pre-trained large language models (LLMs) With DuAl-PT, we propose to learn more context-aware prompts, benefiting from both explicit and implicit context modeling. Empirically, DuAl-PT achieves superior performance on 11 downstream datasets on few-shot recognition and base-to-new generalization.
  arXiv  Detail & Related papers  (2023-09-08T06:51:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.