Visual Odometry with Neuromorphic Resonator Networks

• URL: http://arxiv.org/abs/2209.02000v3
• Date: Wed, 26 Jun 2024 10:17:08 GMT
• Title: Visual Odometry with Neuromorphic Resonator Networks
• Authors: Alpha Renner, Lazar Supic, Andreea Danielescu, Giacomo Indiveri, E. Paxon Frady, Friedrich T. Sommer, Yulia Sandamirskaya,
• Abstract summary: Visual Odometry (VO) is a method to estimate self-motion of a mobile robot using visual sensors. Neuromorphic hardware offers low-power solutions to many vision and AI problems. We present a modular neuromorphic algorithm that achieves state-of-the-art performance on two-dimensional VO tasks.
• Score: 9.903137966539898
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Odometry (VO) is a method to estimate self-motion of a mobile robot using visual sensors. Unlike odometry based on integrating differential measurements that can accumulate errors, such as inertial sensors or wheel encoders, visual odometry is not compromised by drift. However, image-based VO is computationally demanding, limiting its application in use cases with low-latency, -memory, and -energy requirements. Neuromorphic hardware offers low-power solutions to many vision and AI problems, but designing such solutions is complicated and often has to be assembled from scratch. Here we propose to use Vector Symbolic Architecture (VSA) as an abstraction layer to design algorithms compatible with neuromorphic hardware. Building from a VSA model for scene analysis, described in our companion paper, we present a modular neuromorphic algorithm that achieves state-of-the-art performance on two-dimensional VO tasks. Specifically, the proposed algorithm stores and updates a working memory of the presented visual environment. Based on this working memory, a resonator network estimates the changing location and orientation of the camera. We experimentally validate the neuromorphic VSA-based approach to VO with two benchmarks: one based on an event camera dataset and the other in a dynamic scene with a robotic task.

Related papers

• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Retina-inspired Object Motion Segmentation [0.0]
  Dynamic Vision Sensors (DVS) have emerged as a revolutionary technology with a high temporal resolution that far surpasses RGB cameras. This paper introduces a bio-inspired computer vision method that dramatically reduces the number of parameters by a factor of 1000 compared to prior works.
  arXiv  Detail & Related papers  (2024-08-18T12:28:26Z)
• RD-VIO: Robust Visual-Inertial Odometry for Mobile Augmented Reality in Dynamic Environments [55.864869961717424]
  It is typically challenging for visual or visual-inertial odometry systems to handle the problems of dynamic scenes and pure rotation. We design a novel visual-inertial odometry (VIO) system called RD-VIO to handle both of these problems.
  arXiv  Detail & Related papers  (2023-10-23T16:30:39Z)
• Transformer-based model for monocular visual odometry: a video understanding approach [0.9790236766474201]
  We deal with the monocular visual odometry as a video understanding task to estimate the 6-F camera's pose. We contribute by presenting the TS-DoVO model based on on-temporal self-attention mechanisms to extract features from clips and estimate the motions in an end-to-end manner. Our approach achieved competitive state-of-the-art performance compared with geometry-based and deep learning-based methods on the KITTI visual odometry dataset.
  arXiv  Detail & Related papers  (2023-05-10T13:11:23Z)
• Geometric-aware Pretraining for Vision-centric 3D Object Detection [77.7979088689944]
  We propose a novel geometric-aware pretraining framework called GAPretrain. GAPretrain serves as a plug-and-play solution that can be flexibly applied to multiple state-of-the-art detectors. We achieve 46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with a gain of 2.7 and 2.1 points, respectively.
  arXiv  Detail & Related papers  (2023-04-06T14:33:05Z)
• Neuromorphic Visual Scene Understanding with Resonator Networks [11.701553530610973]
  We propose a neuromorphic solution exploiting three key concepts. The framework is based on Vector Architectures with complex-valued vectors. The network is factorized to factorize the non-commutative transforms translation and rotation in visual scenes. A companion paper demonstrates the same approach in real-world application scenarios for machine vision and robotics.
  arXiv  Detail & Related papers  (2022-08-26T22:17:52Z)
• Multitask AET with Orthogonal Tangent Regularity for Dark Object Detection [84.52197307286681]
  We propose a novel multitask auto encoding transformation (MAET) model to enhance object detection in a dark environment. In a self-supervision manner, the MAET learns the intrinsic visual structure by encoding and decoding the realistic illumination-degrading transformation. We have achieved the state-of-the-art performance using synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-05-06T16:27:14Z)
• FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks [77.34726150561087]
  We show how to explore high-dimensional landscape characteristics of neural networks. We generalize observations on small neural networks to more complex systems. An interactive dashboard opens up a number of possible application networks.
  arXiv  Detail & Related papers  (2022-04-09T16:41:53Z)
• Instantaneous Stereo Depth Estimation of Real-World Stimuli with a Neuromorphic Stereo-Vision Setup [4.28479274054892]
  Spiking Neural Network (SNN) architectures for stereo vision have the potential of simplifying the stereo-matching problem. We validate a brain-inspired event-based stereo-matching architecture implemented on a mixed-signal neuromorphic processor with real-world data.
  arXiv  Detail & Related papers  (2021-04-06T14:31:23Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.