Don't Fool Me Twice: Adapting to Adversity in the Wild with Experience-Driven Reasoning

• URL: http://arxiv.org/abs/2605.31119v1
• Date: Fri, 29 May 2026 10:30:02 GMT
• Title: Don't Fool Me Twice: Adapting to Adversity in the Wild with Experience-Driven Reasoning
• Abstract summary: In robotics, dangers and adversity modes are often embodiment-specific and relative to each agent.<n>We propose a continual learning framework for a mobile embodied agent to learn from disturbances and attribute anomalous behaviours to causes.<n>We present four hypotheses and validate them in simulation and on hardware across embodiments and adversity modes.
• Score: 2.3555777010480337
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In robotics, dangers and adversity modes are often embodiment-specific and relative to each agent. A frontier of autonomous mobile robotics is to enable agents to operate effectively in the wild in unseen unstructured environments. A significant challenge in unseen unstructured environments is that it may not be possible to predict all the dangers to the specific robot. Although recent work has used large foundation vision-language models (VLMs) to preemptively predict an exhaustive list of common-sense dangers, it remains difficult to capture possible interaction and embodiment-dependent adversities. We propose a continual learning framework for a mobile embodied agent to learn online from disturbances and attribute anomalous behaviours to causes through semantics, enabling better prediction and planning of the world in the future. Our framework, "Don't Fool Me Twice", first observes disturbances and describes their effects on the robot; this description is augmented with visual context to query a VLM to predict possible causes; the local disturbance is characterized using kernel regression, which allows for efficient, few-shot modeling of transient anomalies. We leverage semantic voxel-centric modeling to estimate epistemic uncertainty, enabling richer downstream recovery by treating interaction-driven disturbances as learnable spatial behaviors. We present four hypotheses and validate them in simulation and on hardware across embodiments and adversity modes.

Related papers

• Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments [19.009627117174947]
  We introduce AnTrap, a comprehensive benchmark that injects dynamic perturbations into agent execution trajectories.<n>We propose a taxonomy organizing real-world anomalies into four layers (State, Thinking, Action and Round) with ten fine-grained subcategories.<n>We also develop a construction pipeline that preserves task solvability while introducing realistic adversarial conditions.
  arXiv  Detail & Related papers  (2026-08-25T05:57:25Z)
• Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation [90.90219129619344]
  This paper presents a novel R-prior-S, Recurrent Geometric-priormodal Policy with Spiking features.<n>To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases.<n>For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network.
  arXiv  Detail & Related papers  (2026-01-13T23:36:30Z)
• UF-RNN: Real-Time Adaptive Motion Generation Using Uncertainty-Driven Foresight Prediction [4.849928323880955]
  Training robots to operate effectively in environments with uncertain states remains a longstanding challenge in robotics.<n>We propose the Uncertainty-driven Foresight Recurrent Neural Network (UF-RNN), a model that combines standard time-series prediction with an active "Foresight" module.<n>UF-RNN exhibits robust adaptation by leveraging self-induced chaotic dynamics in its latent space.<n>These findings suggest that integrating uncertainty-driven foresight into imitation learning pipelines can significantly enhance a robot's ability to handle unpredictable real-world conditions.
  arXiv  Detail & Related papers  (2025-10-11T13:44:20Z)
• Situationally-Aware Dynamics Learning [57.698553219660376]
  We propose a novel framework for online learning of hidden state representations.<n>Our approach explicitly models the influence of unobserved parameters on both transition dynamics and reward structures.<n>Experiments in both simulation and real world reveal significant improvements in data efficiency, policy performance, and the emergence of safer, adaptive navigation strategies.
  arXiv  Detail & Related papers  (2025-05-26T06:40:11Z)
• Adaptive Motion Generation Using Uncertainty-Driven Foresight Prediction [2.2120851074630177]
  Uncertainty of environments has long been a difficult characteristic to handle, when performing real-world robot tasks. This paper extended an existing predictive learning based robot control method, which employ foresight prediction using dynamic internal simulation. The results showed that the proposed model adaptively diverged its motion through interaction with the door, whereas conventional methods failed to stably diverge.
  arXiv  Detail & Related papers  (2024-10-01T15:13:27Z)
• Real-Time Anomaly Detection and Reactive Planning with Large Language Models [18.57162998677491]
  Foundation models, e.g., large language models (LLMs), trained on internet-scale data possess zero-shot capabilities. We present a two-stage reasoning framework that incorporates the judgement regarding potential anomalies into a safe control framework. This enables our monitor to improve the trustworthiness of dynamic robotic systems, such as quadrotors or autonomous vehicles.
  arXiv  Detail & Related papers  (2024-07-11T17:59:22Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance [110.63037190641414]
  We propose to learn congestion patterns explicitly and devise a novel "Sense--Learn--Reason--Predict" framework. By decomposing the learning phases into two stages, a "student" can learn contextual cues from a "teacher" while generating collision-free trajectories. In experiments, we demonstrate that the proposed model is able to generate collision-free trajectory predictions in a synthetic dataset.
  arXiv  Detail & Related papers  (2021-03-26T02:42:33Z)
• Risk-Sensitive Sequential Action Control with Multi-Modal Human Trajectory Forecasting for Safe Crowd-Robot Interaction [55.569050872780224]
  We present an online framework for safe crowd-robot interaction based on risk-sensitive optimal control, wherein the risk is modeled by the entropic risk measure. Our modular approach decouples the crowd-robot interaction into learning-based prediction and model-based control. A simulation study and a real-world experiment show that the proposed framework can accomplish safe and efficient navigation while avoiding collisions with more than 50 humans in the scene.
  arXiv  Detail & Related papers  (2020-09-12T02:02:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.