RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding

• URL: http://arxiv.org/abs/2605.19329v2
• Date: Thu, 21 May 2026 09:49:07 GMT
• Title: RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding
• Abstract summary: We propose RE-VLM, the first dual-stream vision-language model that jointly leverages RGB images and event streams for robust scene understanding.<n>We construct two datasets: PEOD-Chat, targeting illumination-challenged scenes, and RGBE-Chat, covering diverse scenarios.<n>On captioning and VQA benchmarks, RE-VLM consistently outperforms state-of-the-art RGB-only and event-only models.
• Score: 8.729091944523494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conventional vision-language models (VLMs) struggle to interpret scenes captured under adverse conditions (e.g., low light, high dynamic range, or fast motion) because standard RGB images degrade in such environments. Event cameras provide a complementary modality: they asynchronously record per-pixel brightness changes with high temporal resolution and wide dynamic range, preserving motion cues where frames fail. We propose RE-VLM, the first dual-stream vision-language model that jointly leverages RGB images and event streams for robust scene understanding across both normal and challenging conditions. RE-VLM employs parallel RGB and event encoders together with a progressive training strategy that aligns heterogeneous visual features with language. To address the scarcity of RGB-Event-Text supervision, we further propose a graph-driven pipeline that converts synchronized RGB-Event streams into verifiable scene graphs, from which we synthesize captions and question-answer (QA) pairs. To develop and evaluate RE-VLM, we construct two datasets: PEOD-Chat, targeting illumination-challenged scenes, and RGBE-Chat, covering diverse scenarios. On captioning and VQA benchmarks, RE-VLM consistently outperforms state-of-the-art RGB-only and event-only models with comparable parameter counts, with particularly large gains under challenging conditions. These results demonstrate the effectiveness of event-augmented VLMs in achieving robust vision-language understanding across a wide range of real-world environments.

Related papers

• UniRED: Unified RGB-D Video Frame Interpolation with Event Guidance [19.80142530142947]
  High frame-rate RGB-D videos are crucial for a variety of downstream tasks, including motion analysis, dynamic scene understanding, and 3D reconstruction.<n>Existing video methods have achieved strong performance on RGB data, but they are not readily applicable to RGB-D scenarios.<n>We propose a unified multimodal framework for RGB-D video that jointly exploits RGB appearance, depth geometry, and event-based temporal cues.
  arXiv  Detail & Related papers  (2026-06-23T08:03:58Z)
• EventGait: Towards Robust Gait Recognition with Event Streams [50.890621860023]
  Event cameras offer microsecond temporal resolution and high dynamic range, naturally capturing robust dynamic cues and suppressing static noise.<n>We propose textbfEventGait, an end-to-end dual-stream framework that separately models motion and shape while preserving the advantages of events.<n>Our approach sets a new state of the art on both synthesized and real-world event-based gait benchmarks, highlighting the robustness and potential of event-driven gait analysis.
  arXiv  Detail & Related papers  (2026-05-21T08:12:04Z)
• Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor [12.418360718687026]
  Motion blur arises when rapid scene changes occur during the exposure period, collapsing rich intra-exposure motion into a single RGB frame.<n>We propose Spatio-Temporal Difference Guided Deblur Net (STGDNet) to restore structure and color details lost in blurry RGB inputs.<n>Our method outperforms current RGB or event-based approaches in both synthetic CVS dataset and real-world evaluations.
  arXiv  Detail & Related papers  (2026-04-12T09:54:14Z)
• Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models [26.703204607483404]
  Event-MLLM is an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames.<n>We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples.<n>Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines.
  arXiv  Detail & Related papers  (2026-03-29T07:46:32Z)
• EPRBench: A High-Quality Benchmark Dataset for Event Stream Based Visual Place Recognition [54.55914886780534]
  Event stream-based Visual Place Recognition (VPR) is an emerging research direction that offers a compelling solution to the instability of conventional visible-light cameras under challenging conditions such as low illumination, overexposure, and high-speed motion.<n>We introduce EPRBench, a high-quality benchmark specifically designed for event stream-based VPR.<n>EPRBench comprises 10K event sequences and 65K event frames, collected using both handheld and vehicle-mounted setups to comprehensively capture real-world challenges across diverse viewpoints, weather conditions, and lighting scenarios.
  arXiv  Detail & Related papers  (2026-02-13T13:25:05Z)
• IE2Video: Adapting Pretrained Diffusion Models for Event-Based Video Reconstruction [4.452083769109418]
  Event cameras offer sparse, motion-driven sensing with low power consumption.<n>We propose a hybrid capture paradigm that records sparse RGB- sequences alongside continuous event streams.<n>We reconstruct full RGB video offline -- reducing capture power consumption for downstream applications.
  arXiv  Detail & Related papers  (2025-12-04T20:37:45Z)
• DepthVision: Robust Vision-Language Understanding through GAN-Based LiDAR-to-RGB Synthesis [11.976362049118782]
  This letter introduces DepthVision, a framework for multimodal scene understanding.<n>It synthesizes RGB images from sparse LiDAR point clouds using a conditional generative adversarial network (GAN)<n>These synthetic views are then combined with real RGB data using a Luminance-Aware Modality Adaptation (LAMA)
  arXiv  Detail & Related papers  (2025-09-09T07:42:07Z)
• Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithms [58.60058450730943]
  Current sign language translation algorithms predominantly rely on RGB frames, which may be limited by fixed frame rates, variable lighting conditions, and motion blur caused by rapid hand movements.<n>We propose to leverage event streams to assist RGB cameras in capturing gesture data, addressing the various challenges mentioned above.<n>Specifically, we first collect a large-scale RGB-Event sign language translation dataset using the DVS346 camera, which contains 15,676 RGB-Event samples, 15,191 glosses, and covers 2,568 Chinese characters.
  arXiv  Detail & Related papers  (2025-03-09T06:55:46Z)
• Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams [69.65147723239153]
  Volumetric reconstruction of dynamic scenes is an important problem in computer vision.<n>It is especially challenging in poor lighting and with fast motion.<n>We propose the first method totemporally reconstruct a scene from sparse multi-view event streams and sparse RGB frames.
  arXiv  Detail & Related papers  (2024-12-09T18:56:18Z)
• EventGPT: Event Stream Understanding with Multimodal Large Language Models [59.65010502000344]
  Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions.<n>Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better.<n>We introduce EventGPT, the first MLLM for event stream understanding.
  arXiv  Detail & Related papers  (2024-12-01T14:38:40Z)
• SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval [82.51117533271517]
  Previous works typically only encode RGB videos to obtain high-level semantic features. Existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training. We propose a novel sign language representation framework called Semantically Enhanced Dual-Stream.
  arXiv  Detail & Related papers  (2024-07-23T11:31:11Z)
• Revisiting Event-based Video Frame Interpolation [49.27404719898305]
  Dynamic vision sensors or event cameras provide rich complementary information for video frame. estimating optical flow from events is arguably more difficult than from RGB information. We propose a divide-and-conquer strategy in which event-based intermediate frame synthesis happens incrementally in multiple simplified stages.
  arXiv  Detail & Related papers  (2023-07-24T06:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.