Q-learning with temporal memory to navigate turbulence

• URL: http://arxiv.org/abs/2404.17495v2
• Date: Tue, 28 Jan 2025 15:07:43 GMT
• Title: Q-learning with temporal memory to navigate turbulence
• Authors: Marco Rando, Martin James, Alessandro Verri, Lorenzo Rosasco, Agnese Seminara,
• Abstract summary: We consider the problem of olfactory searches in a turbulent environment. We develop a reinforcement learning algorithm using a small set of interpretable olfactory states. We show that two salient features of odor traces, discretized in few olfactory states, are sufficient to learn navigation in a realistic odor plume.
• Score: 47.28605705112213
• License:
• Abstract: We consider the problem of olfactory searches in a turbulent environment. We focus on agents that respond solely to odor stimuli, with no access to spatial perception nor prior information about the odor. We ask whether navigation to a target can be learned robustly within a sequential decision making framework. We develop a reinforcement learning algorithm using a small set of interpretable olfactory states and train it with realistic turbulent odor cues. By introducing a temporal memory, we demonstrate that two salient features of odor traces, discretized in few olfactory states, are sufficient to learn navigation in a realistic odor plume. Performance is dictated by the sparse nature of turbulent odors. An optimal memory exists which ignores blanks within the plume and activates a recovery strategy outside the plume. We obtain the best performance by letting agents learn their recovery strategy and show that it is mostly casting cross wind, similar to behavior observed in flying insects. The optimal strategy is robust to substantial changes in the odor plumes, suggesting minor parameter tuning may be sufficient to adapt to different environments.

Related papers

• Neuromorphic circuit for temporal odor encoding in turbulent environments [0.48748194765816943]
  We investigate Metal-Oxide (MOx) gas sensor recordings of constant airflow-embedded artificial odor plumes. We design a neuromorphic electronic nose front-end circuit for extracting and encoding this feature into analog spikes for gas detection and concentration estimation. The resulting neuromorphic nose could enable data-efficient, real-time robotic plume navigation systems.
  arXiv  Detail & Related papers  (2024-12-28T11:12:18Z)
• Characterizing Requirements Smells [1.565361244756411]
  This paper aims to characterise 12 requirements smells in terms of frequency, severity, and effects. Interview shows that the smell types perceived as most severe are Ambiguity and Verifiability. We also provide a set of six lessons learnt about requirements smells, such as that effects of smells are expected to differ across smell types.
  arXiv  Detail & Related papers  (2024-04-17T06:43:02Z)
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning [125.61772424068903]
  Vision-and-language navigation (VLN) asks an agent to follow a given language instruction to navigate through a real 3D environment. We present a model-agnostic training paradigm, called Progressive Perturbation-aware Contrastive Learning (PROPER) to enhance the generalization ability of existing VLN agents.
  arXiv  Detail & Related papers  (2024-03-09T02:34:13Z)
• Limitations in odour recognition and generalisation in a neuromorphic olfactory circuit [0.07589017023705934]
  We present an odour-learning algorithm that runs on a neuromorphic architecture and is inspired by circuits described in the mammalian olfactory bulb. They assess the algorithm's performance in "rapid online learning and identification" of gaseous odorants and odorless gases.
  arXiv  Detail & Related papers  (2023-09-20T18:00:05Z)
• Emergence of Maps in the Memories of Blind Navigation Agents [68.41901534985575]
  Animal navigation research posits that organisms build and maintain internal spatial representations, or maps, of their environment. We ask if machines -- specifically, artificial intelligence (AI) navigation agents -- also build implicit (or'mental') maps. Unlike animal navigation, we can judiciously design the agent's perceptual system and control the learning paradigm to nullify alternative navigation mechanisms.
  arXiv  Detail & Related papers  (2023-01-30T20:09:39Z)
• Explore before Moving: A Feasible Path Estimation and Memory Recalling Framework for Embodied Navigation [117.26891277593205]
  We focus on the navigation and solve the problem of existing navigation algorithms lacking experience and common sense. Inspired by the human ability to think twice before moving and conceive several feasible paths to seek a goal in unfamiliar scenes, we present a route planning method named Path Estimation and Memory Recalling framework. We show strong experimental results of PEMR on the EmbodiedQA navigation task.
  arXiv  Detail & Related papers  (2021-10-16T13:30:55Z)
• Emergent behavior and neural dynamics in artificial agents tracking turbulent plumes [1.8065361710947974]
  We use deep reinforcement learning to train recurrent neural network (RNN) agents to locate the source of simulated turbulent plumes. Our analyses suggest an intriguing experimentally testable hypothesis for tracking plumes in changing wind direction.
  arXiv  Detail & Related papers  (2021-09-25T20:57:02Z)
• Algorithmic insights on continual learning from fruit flies [22.34773145953582]
  Continual learning in computational systems is challenging due to catastrophic forgetting. We discovered a two layer neural circuit in the fruit fly olfactory system that addresses this challenge. In the first layer, odors are encoded using sparse, high dimensional representations, which reduces memory interference. In the second layer, only the synapses between odor activated neurons and the output neuron associated with the odor are modified during learning.
  arXiv  Detail & Related papers  (2021-07-15T21:28:53Z)
• Pushing it out of the Way: Interactive Visual Navigation [62.296686176988125]
  We study the problem of interactive navigation where agents learn to change the environment to navigate more efficiently to their goals. We introduce the Neural Interaction Engine (NIE) to explicitly predict the change in the environment caused by the agent's actions. By modeling the changes while planning, we find that agents exhibit significant improvements in their navigational capabilities.
  arXiv  Detail & Related papers  (2021-04-28T22:46:41Z)
• Learning Invariant Representations for Reinforcement Learning without Reconstruction [98.33235415273562]
  We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction. Bisimulation metrics quantify behavioral similarity between states in continuous MDPs. We demonstrate the effectiveness of our method at disregarding task-irrelevant information using modified visual MuJoCo tasks.
  arXiv  Detail & Related papers  (2020-06-18T17:59:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.