Show, Don't Tell: Detecting Novel Objects by Watching Human Videos

• URL: http://arxiv.org/abs/2603.12751v1
• Date: Fri, 13 Mar 2026 07:49:46 GMT
• Title: Show, Don't Tell: Detecting Novel Objects by Watching Human Videos
• Abstract summary: "Show, Don't Tell" paradigm enables us to quickly train bespoke detectors tailored to the relevant objects observed in human task demonstrations.<n>We develop an integrated on-robot system to deploy our "Show, Don't Tell" paradigm of automatic dataset creation and novel object-detection on a real-world robot.
• Score: 7.501069961942297
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How can a robot quickly identify and recognize new objects shown to it during a human demonstration? Existing closed-set object detectors frequently fail at this because the objects are out-of-distribution. While open-set detectors (e.g., VLMs) sometimes succeed, they often require expensive and tedious human-in-the-loop prompt engineering to uniquely recognize novel object instances. In this paper, we present a self-supervised system that eliminates the need for tedious language descriptions and expensive prompt engineering by training a bespoke object detector on an automatically created dataset, supervised by the human demonstration itself. In our approach, "Show, Don't Tell," we show the detector the specific objects of interest during the demonstration, rather than telling the detector about these objects via complex language descriptions. By bypassing language altogether, this paradigm enables us to quickly train bespoke detectors tailored to the relevant objects observed in human task demonstrations. We develop an integrated on-robot system to deploy our "Show, Don't Tell" paradigm of automatic dataset creation and novel object-detection on a real-world robot. Empirical results demonstrate that our pipeline significantly outperforms state-of-the-art detection and recognition methods for manipulated objects, leading to improved task completion for the robot.

Related papers

• Slot-Level Robotic Placement via Visual Imitation from Single Human Video [43.71981951957892]
  We address the problem of teaching new tasks to robots using human demonstration videos for repetitive tasks.<n>We propose SLeRP, a modular system that leverages several advanced visual foundation models and a novel slot-level placement detector Slot-Net.
  arXiv  Detail & Related papers  (2025-04-02T17:59:45Z)
• WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model [92.90127398282209]
  This paper investigates the potential of integrating the most recent Large Language Models (LLMs) and existing visual grounding and robotic grasping system. We introduce the WALL-E (Embodied Robotic WAiter load lifting with Large Language model) as an example of this integration. We deploy this LLM-empowered system on the physical robot to provide a more user-friendly interface for the instruction-guided grasping task.
  arXiv  Detail & Related papers  (2023-08-30T11:35:21Z)
• Follow Anything: Open-set detection, tracking, and following in real-time [89.83421771766682]
  We present a robotic system to detect, track, and follow any object in real-time. Our approach, dubbed follow anything'' (FAn), is an open-vocabulary and multimodal model. FAn can be deployed on a laptop with a lightweight (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second.
  arXiv  Detail & Related papers  (2023-08-10T17:57:06Z)
• DetGPT: Detect What You Need via Reasoning [33.00345609506097]
  We introduce a new paradigm for object detection that we call reasoning-based object detection. Unlike conventional object detection methods that rely on specific object names, our approach enables users to interact with the system using natural language instructions. Our proposed method, called DetGPT, leverages state-of-the-art multi-modal models and open-vocabulary object detectors.
  arXiv  Detail & Related papers  (2023-05-23T15:37:28Z)
• Challenges in Visual Anomaly Detection for Mobile Robots [65.53820325712455]
  We consider the task of detecting anomalies for autonomous mobile robots based on vision. We categorize relevant types of visual anomalies and discuss how they can be detected by unsupervised deep learning methods.
  arXiv  Detail & Related papers  (2022-09-22T13:26:46Z)
• Task-Focused Few-Shot Object Detection for Robot Manipulation [1.8275108630751844]
  We develop a manipulation method based solely on detection then introduce task-focused few-shot object detection to learn new objects and settings. In experiments for our interactive approach to few-shot learning, we train a robot to manipulate objects directly from detection (ClickBot)
  arXiv  Detail & Related papers  (2022-01-28T21:52:05Z)
• INVIGORATE: Interactive Visual Grounding and Grasping in Clutter [56.00554240240515]
  INVIGORATE is a robot system that interacts with human through natural language and grasps a specified object in clutter. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. We build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules.
  arXiv  Detail & Related papers  (2021-08-25T07:35:21Z)
• Property-Aware Robot Object Manipulation: a Generative Approach [57.70237375696411]
  In this work, we focus on how to generate robot motion adapted to the hidden properties of the manipulated objects. We explore the possibility of leveraging Generative Adversarial Networks to synthesize new actions coherent with the properties of the object. Our results show that Generative Adversarial Nets can be a powerful tool for the generation of novel and meaningful transportation actions.
  arXiv  Detail & Related papers  (2021-06-08T14:15:36Z)
• Simultaneous Multi-View Object Recognition and Grasping in Open-Ended Domains [0.0]
  We propose a deep learning architecture with augmented memory capacities to handle open-ended object recognition and grasping simultaneously. We demonstrate the ability of our approach to grasp never-seen-before objects and to rapidly learn new object categories using very few examples on-site in both simulation and real-world settings.
  arXiv  Detail & Related papers  (2021-06-03T14:12:11Z)
• Self-supervised object detection from audio-visual correspondence [101.46794879729453]
  We tackle the problem of learning object detectors without supervision. We do not assume image-level class labels, instead we extract a supervisory signal from audio-visual data. We show that our method can learn to detect generic objects that go beyond instruments, such as airplanes and cats.
  arXiv  Detail & Related papers  (2021-04-13T17:59:03Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.