Open-World Distributed Robot Self-Localization with Transferable Visual Vocabulary and Both Absolute and Relative Features

• URL: http://arxiv.org/abs/2109.04569v3
• Date: Wed, 11 Sep 2024 05:25:19 GMT
• Title: Open-World Distributed Robot Self-Localization with Transferable Visual Vocabulary and Both Absolute and Relative Features
• Authors: Mitsuki Yoshida, Ryogo Yamamoto, Daiki Iwata, Kanji Tanaka,
• Abstract summary: This work introduces a new self-localization framework for open-world distributed robot systems. It employs an unsupervised visual vocabulary model that maps to multimodal, lightweight, and transferable visual features. All features are learned and recognized using a lightweight graph neural network and scene graph.
• Score: 1.3499500088995464
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual robot self-localization is a fundamental problem in visual robot navigation and has been studied across various problem settings, including monocular and sequential localization. However, many existing studies focus primarily on single-robot scenarios, with limited exploration into general settings involving diverse robots connected through wireless networks with constrained communication capacities, such as open-world distributed robot systems. In particular, issues related to the transfer and sharing of key knowledge, such as visual descriptions and visual vocabulary, between robots have been largely neglected. This work introduces a new self-localization framework designed for open-world distributed robot systems that maintains state-of-the-art performance while offering two key advantages: (1) it employs an unsupervised visual vocabulary model that maps to multimodal, lightweight, and transferable visual features, and (2) the visual vocabulary itself is a lightweight and communication-friendly model. Although the primary focus is on encoding monocular view images, the framework can be easily extended to sequential localization applications. By utilizing complementary similarity-preserving features -- both absolute and relative -- the framework meets the requirements for being unsupervised, multimodal, lightweight, and transferable. All features are learned and recognized using a lightweight graph neural network and scene graph. The effectiveness of the proposed method is validated in both passive and active self-localization scenarios.

Related papers

• MGCR-Net:Multimodal Graph-Conditioned Vision-Language Reconstruction Network for Remote Sensing Change Detection [55.702662643521265]
  We propose the multimodal graph-conditioned vision-language reconstruction network (MGCR-Net) to explore the semantic interaction capabilities of multimodal data.<n> Experimental results on four public datasets demonstrate that MGCR achieves superior performance compared to mainstream CD methods.
  arXiv  Detail & Related papers  (2025-08-03T02:50:08Z)
• Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding [85.63710017456792]
  FuSe is a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities. We show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound. Experiments in the real world show that FuSeis able to increase success rates by over 20% compared to all considered baselines.
  arXiv  Detail & Related papers  (2025-01-08T18:57:33Z)
• OV-HHIR: Open Vocabulary Human Interaction Recognition Using Cross-modal Integration of Large Language Models [4.831029473163422]
  We propose an open vocabulary human-to-human interaction recognition framework. We generate open-ended textual descriptions of both seen and unseen human interactions in open-world settings. Our method outperforms traditional fixed-vocabulary classification systems and existing cross-modal language models for video understanding.
  arXiv  Detail & Related papers  (2024-12-31T13:22:00Z)
• Context-Aware Command Understanding for Tabletop Scenarios [1.7082212774297747]
  This paper presents a novel hybrid algorithm designed to interpret natural human commands in tabletop scenarios. By integrating multiple sources of information, including speech, gestures, and scene context, the system extracts actionable instructions for a robot. We discuss the strengths and limitations of the system, with particular focus on how it handles multimodal command interpretation.
  arXiv  Detail & Related papers  (2024-10-08T20:46:39Z)
• DeepInteraction++: Multi-Modality Interaction for Autonomous Driving [80.8837864849534]
  We introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout. DeepInteraction++ is a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:04:21Z)
• Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts [21.249837293326497]
  Generalizable reward function is central to reinforcement learning and planning for robots. This paper transfers video-language models with robust generalization into a language-conditioned reward function. Our model shows outstanding generalization to new environments and new instructions for robot planning and reinforcement learning.
  arXiv  Detail & Related papers  (2024-07-20T13:22:59Z)
• Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
  Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks. We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders. Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
  arXiv  Detail & Related papers  (2023-10-04T17:59:38Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models. Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning. Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)
• PaLM-E: An Embodied Multimodal Language Model [101.29116156731762]
  We propose embodied language models to incorporate real-world continuous sensor modalities into language models. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks. Our largest model, PaLM-E-562B with 562B parameters, is a visual-language generalist with state-of-the-art performance on OK-VQA.
  arXiv  Detail & Related papers  (2023-03-06T18:58:06Z)
• Enhancing Interpretability and Interactivity in Robot Manipulation: A Neurosymbolic Approach [0.0]
  We present a neurosymbolic architecture for coupling language-guided visual reasoning with robot manipulation. A non-expert human user can prompt the robot using unconstrained natural language, providing a referring expression (REF), a question (VQA) or a grasp action instruction. We generate a 3D vision-and-language synthetic dataset of tabletop scenes in a simulation environment to train our approach and perform extensive evaluations in both synthetic and real-world scenes.
  arXiv  Detail & Related papers  (2022-10-03T12:21:45Z)
• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections [104.14624185375897]
  mPLUG is a new vision-language foundation model for both cross-modal understanding and generation. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering.
  arXiv  Detail & Related papers  (2022-05-24T11:52:06Z)
• Enhancing Multi-Robot Perception via Learned Data Association [37.866254392010454]
  We address the multi-robot collaborative perception problem, specifically in the context of multi-view infilling for distributed semantic segmentation. We propose the Multi-Agent Infilling Network: an neural architecture that can be deployed to each agent in a robotic swarm. Specifically, each robot is in charge of locally encoding and decoding visual information, and an neural mechanism allows for an uncertainty-aware and context-based exchange of intermediate features.
  arXiv  Detail & Related papers  (2021-07-01T22:45:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.