MAMMOTH: A Multi-Modal End-to-End Policy for Off-Road Mobility Robust to Missing Modality

• URL: http://arxiv.org/abs/2607.12965v1
• Date: Tue, 14 Jul 2026 17:04:47 GMT
• Title: MAMMOTH: A Multi-Modal End-to-End Policy for Off-Road Mobility Robust to Missing Modality
• Abstract summary: We introduce MAMMOTH, a unified end-to-end navigation policy for robust off-road visual-goal-conditioned navigation and undirected exploration.<n>Specifically, MAMMOTH efficiently fuses multi-modal observations (RGB, Thermal, 3D Pointcloud and Ego Velocity) and is trained with a modality dropout scheme.<n>Our results demonstrate superior performance, with significant improvements in collision avoidance, terrain-aware planning and generalization to missing modalities.
• Score: 5.507970541170981
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reliable autonomous navigation in unstructured off-road environments remains a critical unsolved challenge due to extreme terrain diversity, drastic illumination variations and acute sensor degradation. Recent developments have approached the problem as a traversability costmap estimation or visual navigation task. However, many exhibit heavy reliance on RGB modality, leading to poor performance in varied illumination such as glares, shadows or low ambient light. Achieving robust generalization in such conditions requires integrating modalities that provide supplementary scene information. Such multi-modal methods suffer from a rigid dependency on the presence of near-perfect sensor inputs, leaving them unable to robustly handle sensor degradation or individual modality failure. To address these limitations, we introduce MAMMOTH (MAsking Multi-Modal inputs for Off-road Traversability Heuristic-informed navigation), a unified end-to-end navigation policy for robust off-road visual-goal-conditioned navigation and undirected exploration. Specifically, MAMMOTH efficiently fuses multi-modal observations (RGB, Thermal, 3D Pointcloud and Ego Velocity) and is trained with a modality dropout scheme, enabling it to generalize to missing modalities at inference time. Furthermore, we employ a diffusion policy to learn the joint conditional probability distribution of physically-grounded trajectories and a intrinsic traversability heuristic. MAMMOTH utilizes this heuristic to prefer safer, smoother trajectories. We validate MAMMOTH through extensive real-world robot experiments in distinct off-road environments, including night-time operation. Our results demonstrate superior performance, with significant improvements in collision avoidance, terrain-aware planning and generalization to missing modalities. The code and dataset used for this work will be made publicly available.

Related papers

• CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Driving [15.316817374554828]
  Cruise is a novel uncertainty-aware cross-modal sensor fusion framework.<n>It integrates a vision-language model (VLM)-guided UQ module that generates fine-grained, pixel-level uncertainty estimates.<n>We introduce a dynamic adaptive mechanism that explicitly models and captures cross-modal dependencies.
  arXiv  Detail & Related papers  (2026-08-10T07:17:00Z)
• UNSEEN: Uncertainty-aware Navigation via Sparse Estimation in Unknown Environments [5.893948403053389]
  Vision-only navigation offers a lightweight alternative, but its performance degrades severely under motion blur, low texture, and illumination changes.<n>We present UNSEEN, a unified uncertainty- and perception-aware navigation framework that explicitly couples localization, mapping, and planning using only a front-mounted camera.
  arXiv  Detail & Related papers  (2026-06-18T07:54:07Z)
• A CLIP-based Uncertainty Modal Modeling (UMM) Framework for Pedestrian Re-Identification in Autonomous Driving [6.223368492604449]
  Uncertainty Modal Modeling (UMM) framework integrates a multimodal token mapper, synthetic modality augmentation strategy, and cross-modal cue interactive learner.<n>UMM achieves strong robustness, generalization, and computational efficiency under uncertain modality conditions.
  arXiv  Detail & Related papers  (2025-08-15T04:50:27Z)
• What You Have is What You Track: Adaptive and Robust Multimodal Tracking [72.92244578461869]
  We present the first comprehensive study on tracker performance with temporally incomplete multimodal data.<n>Our model achieves SOTA performance across 9 benchmarks, excelling in both conventional complete and missing modality settings.
  arXiv  Detail & Related papers  (2025-07-08T11:40:21Z)
• TOMD: A Trail-based Off-road Multimodal Dataset for Traversable Pathway Segmentation under Challenging Illumination Conditions [17.019567722324666]
  We introduce the Trail-based Off-road Multimodal dataset (TOMD), a comprehensive dataset specifically designed for such environments.<n>TOMD features high-fidelity multimodal sensor data -- including 128-channel LiDAR, stereo imagery, IMU, and illumination measurements.<n>We also propose a dynamic multiscale data fusion model for accurate traversable pathway prediction.
  arXiv  Detail & Related papers  (2025-06-24T23:58:44Z)
• Easy-Poly: A Easy Polyhedral Framework For 3D Multi-Object Tracking [23.40561503456164]
  We present Easy-Poly, a real-time, filter-based 3D MOT framework for multiple object categories.<n>Results show that Easy-Poly outperforms state-of-the-art methods such as Poly-MOT and Fast-Poly.<n>These findings highlight Easy-Poly's adaptability and robustness in diverse scenarios.
  arXiv  Detail & Related papers  (2025-02-25T04:01:25Z)
• PFSD: A Multi-Modal Pedestrian-Focus Scene Dataset for Rich Tasks in Semi-Structured Environments [73.80718037070773]
  We present the multi-modal Pedestrian-Focused Scene dataset, rigorously annotated in semi-structured scenes with the format of nuScenes.<n>We also propose a novel Hybrid Multi-Scale Fusion Network (HMFN) to detect pedestrians in densely populated and occluded scenarios.
  arXiv  Detail & Related papers  (2025-02-21T09:57:53Z)
• Multi-Modality Driven LoRA for Adverse Condition Depth Estimation [61.525312117638116]
  We propose Multi-Modality Driven LoRA (MMD-LoRA) for Adverse Condition Depth Estimation.<n>It consists of two core components: Prompt Driven Domain Alignment (PDDA) and Visual-Text Consistent Contrastive Learning (VTCCL)<n>It achieves state-of-the-art performance on the nuScenes and Oxford RobotCar datasets.
  arXiv  Detail & Related papers  (2024-12-28T14:23:58Z)
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models [114.69732301904419]
  We present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. Our approach demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations.
  arXiv  Detail & Related papers  (2023-10-26T17:56:35Z)
• SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation [152.60469768559878]
  SHIFT is the largest multi-task synthetic dataset for autonomous driving. It presents discrete and continuous shifts in cloudiness, rain and fog intensity, time of day, and vehicle and pedestrian density. Our dataset and benchmark toolkit are publicly available at www.vis.xyz/shift.
  arXiv  Detail & Related papers  (2022-06-16T17:59:52Z)
• Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning [59.19469551774703]
  Drone-based vehicle detection aims at finding the vehicle locations and categories in an aerial image. We construct a large-scale drone-based RGB-Infrared vehicle detection dataset, termed DroneVehicle. Our DroneVehicle collects 28, 439 RGB-Infrared image pairs, covering urban roads, residential areas, parking lots, and other scenarios from day to night.
  arXiv  Detail & Related papers  (2020-03-05T05:29:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.