Dynamic-Dark SLAM: RGB-Thermal Cooperative Robot Vision Strategy for Multi-Person Tracking in Both Well-Lit and Low-Light Scenes

• URL: http://arxiv.org/abs/2503.12768v3
• Date: Mon, 04 Aug 2025 08:04:09 GMT
• Title: Dynamic-Dark SLAM: RGB-Thermal Cooperative Robot Vision Strategy for Multi-Person Tracking in Both Well-Lit and Low-Light Scenes
• Abstract summary: We propose a cooperative MPT system that utilizes co-located RGB and thermal cameras.<n> Evaluation experiments demonstrate that the thermal tracker performs robustly in both bright and dark environments.<n>This study takes a first step toward a new Human-Only SLAM paradigm, Dynamic-Dark SLAM''
• Score: 1.1874952582465603
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In robot vision, thermal cameras hold great potential for recognizing humans even in complete darkness. However, their application to multi-person tracking (MPT) has been limited due to data scarcity and the inherent difficulty of distinguishing individuals. In this study, we propose a cooperative MPT system that utilizes co-located RGB and thermal cameras, where pseudo-annotations (bounding boxes and person IDs) are used to train both RGB and thermal trackers. Evaluation experiments demonstrate that the thermal tracker performs robustly in both bright and dark environments. Moreover, the results suggest that a tracker-switching strategy -- guided by a binary brightness classifier -- is more effective for information integration than a tracker-fusion approach. As an application example, we present an image change pattern recognition (ICPR) method, the ``human-as-landmark,'' which combines two key properties: the thermal recognizability of humans in dark environments and the rich landmark characteristics -- appearance, geometry, and semantics -- of static objects (occluders). Whereas conventional SLAM focuses on mapping static landmarks in well-lit environments, the present study takes a first step toward a new Human-Only SLAM paradigm, ``Dynamic-Dark SLAM,'' which aims to map even dynamic landmarks in complete darkness. Additionally, this study demonstrates that knowledge transfer between thermal and depth modalities enables reliable person tracking using low-resolution 3D LiDAR data without RGB input, contributing an important advance toward cross-robot SLAM systems.

Related papers

• A Dual-Stream Transformer Architecture for Illumination-Invariant TIR-LiDAR Person Tracking [0.7734726150561086]
  This paper proposes a novel Thermal-Infrared and Depth (TIR-D) tracking architecture that leverages the standard sensor suite of SLAM-capable robots.<n>We introduce a sequential knowledge transfer strategy that evolves structural priors from a large-scale thermal-trained model into the TIR-D domain.<n> Experimental results demonstrate that our proposed TIR-D tracker achieves superior performance, with an Average Overlap (AO) of 0.700 and a Success Rate (SR) of 58.7%.
  arXiv  Detail & Related papers  (2026-04-01T01:20:00Z)
• Dropping the D: RGB-D SLAM Without the Depth Sensor [16.83416267639945]
  We present DropD-SLAM, a real-time monocular SLAM system that achieves RGB-D-level accuracy without relying on depth sensors.<n>The system replaces active depth input with three pretrained vision modules.<n>On the TUM RGB-D benchmark, DropD-SLAM attains 7.4 cm mean ATE on static sequences and 1.8 cm on dynamic sequences.
  arXiv  Detail & Related papers  (2025-10-07T17:59:30Z)
• Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm [103.36490810025752]
  Existing multi-modal object tracking approaches primarily focus on dual-modal paradigms, such as RGB-Depth or RGB-Thermal.<n>This work introduces a novel multi-modal tracking task that leverages three complementary modalities, including visible RGB, Depth (D), and Thermal Infrared (TIR)<n>We propose a novel multi-modal tracker, dubbed RDTTrack, which integrates tri-modal information for robust tracking by leveraging a pretrained RGB-only tracking model.
  arXiv  Detail & Related papers  (2025-09-29T13:05:15Z)
• Human Activity Recognition using RGB-Event based Sensors: A Multi-modal Heat Conduction Model and A Benchmark Dataset [65.76480665062363]
  Human Activity Recognition primarily relied on traditional RGB cameras to achieve high-performance activity recognition. Challenges in real-world scenarios, such as insufficient lighting and rapid movements, inevitably degrade the performance of RGB cameras. In this work, we rethink human activity recognition by combining the RGB and event cameras.
  arXiv  Detail & Related papers  (2025-04-08T09:14:24Z)
• Multi-modal Multi-platform Person Re-Identification: Benchmark and Method [58.59888754340054]
  MP-ReID is a novel dataset designed specifically for multi-modality and multi-platform ReID. This benchmark compiles data from 1,930 identities across diverse modalities, including RGB, infrared, and thermal imaging. We introduce Uni-Prompt ReID, a framework with specific-designed prompts, tailored for cross-modality and cross-platform scenarios.
  arXiv  Detail & Related papers  (2025-03-21T12:27:49Z)
• RGB-Thermal Infrared Fusion for Robust Depth Estimation in Complex Environments [0.0]
  This paper proposes a novel multimodal depth estimation model, RTFusion, which enhances depth estimation accuracy and robustness.<n>The model incorporates a unique fusion mechanism, EGFusion, consisting of the Mutual Complementary Attention (MCA) module for cross-modal feature alignment.<n>Experiments on the MS2 and ViViD++ datasets demonstrate that the proposed model consistently produces high-quality depth maps.
  arXiv  Detail & Related papers  (2025-03-05T01:35:14Z)
• SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking [19.50096632818305]
  Multimodal Visual Object Tracking (VOT) has recently gained significant attention due to its robustness. Recent studies have utilized prompt tuning to transfer pre-trained RGB-based trackers to multimodal data. We propose a novel symmetric multimodal tracking framework called SDSTrack.
  arXiv  Detail & Related papers  (2024-03-24T04:15:50Z)
• Egocentric RGB+Depth Action Recognition in Industry-Like Settings [50.38638300332429]
  Our work focuses on recognizing actions from egocentric RGB and Depth modalities in an industry-like environment. Our framework is based on the 3D Video SWIN Transformer to encode both RGB and Depth modalities effectively. Our method also secured first place at the multimodal action recognition challenge at ICIAP 2023.
  arXiv  Detail & Related papers  (2023-09-25T08:56:22Z)
• Multi-Modal Hybrid Learning and Sequential Training for RGB-T Saliency Detection [10.589062261564631]
  RGB-T saliency detection has emerged as an important computer vision task, identifying conspicuous objects in challenging scenes such as dark environments. Existing methods neglect the characteristics of cross-modal features and rely solely on network structures to fuse RGB and thermal features. We first propose a Multi-Modal Hybrid loss (MMHL) that comprises supervised and self-supervised loss functions.
  arXiv  Detail & Related papers  (2023-09-13T20:47:29Z)
• Learning Dual-Fused Modality-Aware Representations for RGBD Tracking [67.14537242378988]
  Compared with the traditional RGB object tracking, the addition of the depth modality can effectively solve the target and background interference. Some existing RGBD trackers use the two modalities separately and thus some particularly useful shared information between them is ignored. We propose a novel Dual-fused Modality-aware Tracker (termed DMTracker) which aims to learn informative and discriminative representations of the target objects for robust RGBD tracking.
  arXiv  Detail & Related papers  (2022-11-06T07:59:07Z)
• Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline [80.13652104204691]
  In this paper, we construct a large-scale benchmark with high diversity for visible-thermal UAV tracking (VTUAV) We provide a coarse-to-fine attribute annotation, where frame-level attributes are provided to exploit the potential of challenge-specific trackers. In addition, we design a new RGB-T baseline, named Hierarchical Multi-modal Fusion Tracker (HMFT), which fuses RGB-T data in various levels.
  arXiv  Detail & Related papers  (2022-04-08T15:22:33Z)
• MFGNet: Dynamic Modality-Aware Filter Generation for RGB-T Tracking [72.65494220685525]
  We propose a new dynamic modality-aware filter generation module (named MFGNet) to boost the message communication between visible and thermal data. We generate dynamic modality-aware filters with two independent networks. The visible and thermal filters will be used to conduct a dynamic convolutional operation on their corresponding input feature maps respectively. To address issues caused by heavy occlusion, fast motion, and out-of-view, we propose to conduct a joint local and global search by exploiting a new direction-aware target-driven attention mechanism.
  arXiv  Detail & Related papers  (2021-07-22T03:10:51Z)
• Graph-based Thermal-Inertial SLAM with Probabilistic Neural Networks [38.35547654117047]
  We propose the first complete thermal-inertial SLAM system which combines neural abstraction in the SLAM front end with robust pose graph optimization in the SLAM back end. Our key strategies to successfully model this encoding from thermal imagery are the usage of normalized 14-bit radiometric data, the incorporation of hallucinated visual (RGB) features, and the inclusion of feature selection to estimate the MDN parameters.
  arXiv  Detail & Related papers  (2021-04-15T01:39:15Z)
• Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo [61.6260594326246]
  We introduce an efficient fully-convolutional architecture that can leverage both spatial and photometric context simultaneously. Using separable 4D convolutions and 2D heat-maps reduces the size and makes more efficient.
  arXiv  Detail & Related papers  (2021-03-22T18:06:58Z)
• Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking [85.333260415532]
  We develop a novel late fusion method to infer the fusion weight maps of both RGB and thermal (T) modalities. When the appearance cue is unreliable, we take motion cues into account to make the tracker robust. Numerous results on three recent RGB-T tracking datasets show that the proposed tracker performs significantly better than other state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2020-07-04T08:11:33Z)
• Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning [59.19469551774703]
  Drone-based vehicle detection aims at finding the vehicle locations and categories in an aerial image. We construct a large-scale drone-based RGB-Infrared vehicle detection dataset, termed DroneVehicle. Our DroneVehicle collects 28, 439 RGB-Infrared image pairs, covering urban roads, residential areas, parking lots, and other scenarios from day to night.
  arXiv  Detail & Related papers  (2020-03-05T05:29:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.