Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2603.27558v1
• Date: Sun, 29 Mar 2026 07:46:32 GMT
• Title: Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models
• Abstract summary: Event-MLLM is an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames.<n>We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples.<n>Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines.
• Score: 26.703204607483404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator - a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion - and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination.

Related papers

• Event-Illumination Collaborative Low-light Image Enhancement with a High-resolution Real-world Dataset [74.66758770936579]
  EIC-LIE is an event-illumination collaborative LIE framework.<n>EIC-LIE outperforms state-of-the-art methods on five real-world and synthetic datasets.
  arXiv  Detail & Related papers  (2026-05-21T08:55:34Z)
• RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding [8.729091944523494]
  We propose RE-VLM, the first dual-stream vision-language model that jointly leverages RGB images and event streams for robust scene understanding.<n>We construct two datasets: PEOD-Chat, targeting illumination-challenged scenes, and RGBE-Chat, covering diverse scenarios.<n>On captioning and VQA benchmarks, RE-VLM consistently outperforms state-of-the-art RGB-only and event-only models.
  arXiv  Detail & Related papers  (2026-05-19T04:07:37Z)
• Diffusion-Based Low-Light Image Enhancement with Color and Luminance Priors [13.688097246812042]
  Low-light images often suffer from low contrast, noise, and color distortion, degrading visual quality and impairing downstream vision tasks.<n>We propose a novel conditional diffusion framework for low-light image enhancement that incorporates a Structured Control Embedding Module (SCEM)<n>SCEM decomposes a low-light image into four informative components including illumination, illumination-invariant features, shadow priors, and color-invariant cues.
  arXiv  Detail & Related papers  (2026-02-27T22:15:27Z)
• DIME-Net: A Dual-Illumination Adaptive Enhancement Network Based on Retinex and Mixture-of-Experts [7.6894262288762665]
  We propose a dual-illumination enhancement framework called DIME-Net.<n>By integrating Retinex theory, this module effectively performs enhancement tailored to both low-light and backlit images.<n>We show that DIME-Net achieves competitive performance on both synthetic and real-world low-light and backlit datasets without any retraining.
  arXiv  Detail & Related papers  (2025-08-19T15:17:47Z)
• SAIGFormer: A Spatially-Adaptive Illumination-Guided Network for Low-Light Image Enhancement [58.79901582809091]
  Recent Transformer-based low-light enhancement methods have made promising progress in recovering global illumination.<n>Recent Transformer-based low-light enhancement methods have made promising progress in recovering global illumination.<n>We present a Spatially-Adaptive Illumination-Guided Transformer framework that enables accurate illumination restoration.
  arXiv  Detail & Related papers  (2025-07-21T11:38:56Z)
• Light of Normals: Unified Feature Representation for Universal Photometric Stereo [69.95514862547174]
  Current encoders cannot guarantee that illumination and normal information are decoupled.<n>We introduce LINO UniPS with two key components: (i) Light Register Tokens with light alignment supervision to aggregate point, direction, and environment lights.<n>We also introduce PS-Verse, a large-scale synthetic dataset graded by geometric complexity and lighting diversity.
  arXiv  Detail & Related papers  (2025-06-23T17:53:11Z)
• Event-Driven Dynamic Scene Depth Completion [50.01494043834177]
  EventDC is the first event-driven depth completion framework.<n>It consists of two key components: Event-Modulated Alignment (EMA) and Local Depth Filtering (LDF)
  arXiv  Detail & Related papers  (2025-05-19T16:02:37Z)
• EventGPT: Event Stream Understanding with Multimodal Large Language Models [59.65010502000344]
  Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions.<n>Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better.<n>We introduce EventGPT, the first MLLM for event stream understanding.
  arXiv  Detail & Related papers  (2024-12-01T14:38:40Z)
• Chasing Day and Night: Towards Robust and Efficient All-Day Object Detection Guided by an Event Camera [8.673063170884591]
  EOLO is a novel object detection framework that achieves robust and efficient all-day detection by fusing both RGB and event modalities. Our EOLO framework is built based on a lightweight spiking neural network (SNN) to efficiently leverage the asynchronous property of events.
  arXiv  Detail & Related papers  (2023-09-17T15:14:01Z)
• Diving into Darkness: A Dual-Modulated Framework for High-Fidelity Super-Resolution in Ultra-Dark Environments [51.58771256128329]
  This paper proposes a specialized dual-modulated learning framework that attempts to deeply dissect the nature of the low-light super-resolution task. We develop Illuminance-Semantic Dual Modulation (ISDM) components to enhance feature-level preservation of illumination and color details. Comprehensive experiments showcases the applicability and generalizability of our approach to diverse and challenging ultra-low-light conditions.
  arXiv  Detail & Related papers  (2023-09-11T06:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.