SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine

• URL: http://arxiv.org/abs/2601.12010v1
• Date: Sat, 17 Jan 2026 11:25:55 GMT
• Title: SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine
• Abstract summary: RefAV is an end-to-end framework that uses large language models (LLMs) to spatially and temporally localize scenarios.<n>SMc2f is a pipeline that employs vision-language models (VLMs) for coarse image-text filtering.<n> Experiments on public datasets demonstrate substantial gains in both retrieval quality and efficiency.
• Score: 8.662817298688147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The safety validation of autonomous robotic vehicles hinges on systematically testing their planning and control stacks against rare, safety-critical scenarios. Mining these long-tail events from massive real-world driving logs is therefore a critical step in the robotic development lifecycle. The goal of the Scenario Mining task is to retrieve useful information to enable targeted re-simulation, regression testing, and failure analysis of the robot's decision-making algorithms. RefAV, introduced by the Argoverse team, is an end-to-end framework that uses large language models (LLMs) to spatially and temporally localize scenarios described in natural language. However, this process performs retrieval on trajectory labels, ignoring the direct connection between natural language and raw RGB images, which runs counter to the intuition of video retrieval; it also depends on the quality of upstream 3D object detection and tracking. Further, inaccuracies in trajectory data lead to inaccuracies in downstream spatial and temporal localization. To address these issues, we propose Robust Scenario Mining for Robotic Autonomy from Coarse to Fine (SMc2f), a coarse-to-fine pipeline that employs vision-language models (VLMs) for coarse image-text filtering, builds a database of successful mining cases on top of RefAV and automatically retrieves exemplars to few-shot condition the LLM for more robust retrieval, and introduces text-trajectory contrastive learning to pull matched pairs together and push mismatched pairs apart in a shared embedding space, yielding a fine-grained matcher that refines the LLM's candidate trajectories. Experiments on public datasets demonstrate substantial gains in both retrieval quality and efficiency.

Related papers

• DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation [7.872843554385665]
  DF$3$ injects learnable spatial queries into the terminal blocks of a frozen vision foundation model to extract future state representations directly.<n>Experiments on public benchmarks and zero-shot deployment in a robotic simulator demonstrate that DF$3$ achieves performance comparable to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-08-03T16:08:59Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• Spatial Atlas: Compute-Grounded Reasoning for Spatial-Aware Research Agent Benchmarks [1.5229257192293202]
  We introduce compute-grounded reasoning (CGR) for spatial-aware research agents.<n>CGR is a design paradigm for spatial-aware research agents in which every answerable sub-problem is resolved by deterministic computation.<n>We show that CGR yields competitive accuracy while maintaining interpretability.
  arXiv  Detail & Related papers  (2026-04-13T22:22:07Z)
• Advancing Multi-Robot Networks via MLLM-Driven Sensing, Communication, and Computation: A Comprehensive Survey [35.94145527541823]
  Survey examines how integrated design paves the way for multi-robot coordination under MLLM guidance.<n>We present four end-to-end demonstrations (sense -> communicate -> compute -> act)<n>We emphasize system-level metrics -- payload, latency, and success -- to show why R2X orchestration outperforms purely on-device baselines.
  arXiv  Detail & Related papers  (2026-03-31T06:26:53Z)
• Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detection [53.988759250627425]
  HeROD (Heuristic-inspired ROD) is a lightweight, model-agnostic framework that injects explicit, interpretable spatial and semantic reasonings.<n>HeROD consistently outperforms strong grounding baselines in scarcelabel regimes.
  arXiv  Detail & Related papers  (2026-03-25T10:33:22Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• Detect Anything via Next Point Prediction [51.55967987350882]
  Rex- Omni is a 3B-scale MLLM that achieves state-of-the-art object perception performance.<n>On benchmarks like COCO and LVIS, Rex- Omni attains performance comparable to or exceeding regression-based models.
  arXiv  Detail & Related papers  (2025-10-14T17:59:54Z)
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting [92.57796055887995]
  We introduce ECHO, a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents.<n> ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts.<n>We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation.
  arXiv  Detail & Related papers  (2025-10-11T18:11:09Z)
• DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models [60.713908578319256]
  We propose Direct Discrepancy Learning (DDL) to optimize the detector with task-oriented knowledge.<n>Built upon this, we introduce DetectAnyLLM, a unified detection framework that achieves state-of-the-art MGTD performance.<n>MIRAGE samples human-written texts from 10 corpora across 5 text-domains, which are then re-generated or revised using 17 cutting-edge LLMs.
  arXiv  Detail & Related papers  (2025-09-15T10:59:57Z)
• Context-based Motion Retrieval using Open Vocabulary Methods for Autonomous Driving [0.5249805590164902]
  We propose a novel context-aware motion retrieval framework to support targeted evaluation of autonomous driving systems in diverse, human-centered scenarios.<n>Our approach outperforms state-of-the-art models by up to 27.5% accuracy in motion-context retrieval, when evaluated on the WayMoCo dataset.
  arXiv  Detail & Related papers  (2025-08-01T12:41:52Z)
• RefAV: Towards Planning-Centric Scenario Mining [45.37155349405482]
  Traditional scenario mining techniques are error-prone and prohibitively time-consuming.<n>We introduce RefAV, a large-scale dataset of 10,000 diverse natural language queries.<n>We find that naively structured off-the-shelf-Ms yields poor performance.
  arXiv  Detail & Related papers  (2025-05-27T10:14:35Z)
• Outlier-Robust Long-Term Robotic Mapping Leveraging Ground Segmentation [1.7948767405202701]
  I propose a robust long-term robotic mapping system that can work out of the box. I propose (i) fast and robust ground segmentation to reject the presence of outliers. I propose (ii)-robust registration with ground segmentation that encompasses the presence of gross outliers.
  arXiv  Detail & Related papers  (2024-05-18T04:56:15Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• SABER: Data-Driven Motion Planner for Autonomously Navigating Heterogeneous Robots [112.2491765424719]
  We present an end-to-end online motion planning framework that uses a data-driven approach to navigate a heterogeneous robot team towards a global goal. We use model predictive control (SMPC) to calculate control inputs that satisfy robot dynamics, and consider uncertainty during obstacle avoidance with chance constraints. recurrent neural networks are used to provide a quick estimate of future state uncertainty considered in the SMPC finite-time horizon solution. A Deep Q-learning agent is employed to serve as a high-level path planner, providing the SMPC with target positions that move the robots towards a desired global goal.
  arXiv  Detail & Related papers  (2021-08-03T02:56:21Z)
• Exploring Dynamic Context for Multi-path Trajectory Prediction [33.66335553588001]
  We propose a novel framework, named Dynamic Context Network (DCENet) In our framework, the spatial context between agents is explored by using self-attention architectures. A set of future trajectories for each agent is predicted conditioned on the learned spatial-temporal context.
  arXiv  Detail & Related papers  (2020-10-30T13:39:20Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.