Towards smart and adaptive agents for active sensing on edge devices

• URL: http://arxiv.org/abs/2501.06262v1
• Date: Thu, 09 Jan 2025 13:27:02 GMT
• Title: Towards smart and adaptive agents for active sensing on edge devices
• Authors: Devendra Vyas, Miguel de Prado, Tim Verbelen,
• Abstract summary: TinyML has made deploying deep learning models on low-power edge devices feasible. Deep learning's scaling laws cannot be applied when deploying on the Edge. This paper presents a smart agentic system capable of performing on-device perception and planning.
• Score: 4.2534846356464815
• License:
• Abstract: TinyML has made deploying deep learning models on low-power edge devices feasible, creating new opportunities for real-time perception in constrained environments. However, the adaptability of such deep learning methods remains limited to data drift adaptation, lacking broader capabilities that account for the environment's underlying dynamics and inherent uncertainty. Deep learning's scaling laws, which counterbalance this limitation by massively up-scaling data and model size, cannot be applied when deploying on the Edge, where deep learning limitations are further amplified as models are scaled down for deployment on resource-constrained devices. This paper presents a smart agentic system capable of performing on-device perception and planning, enabling active sensing on the edge. By incorporating active inference into our solution, our approach extends beyond deep learning capabilities, allowing the system to plan in dynamic environments while operating in real time with a modest total model size of 2.3 MB. We showcase our proposed system by creating and deploying a saccade agent connected to an IoT camera with pan and tilt capabilities on an NVIDIA Jetson embedded device. The saccade agent controls the camera's field of view following optimal policies derived from the active inference principles, simulating human-like saccadic motion for surveillance and robotics applications.

Related papers

• EdgeMLOps: Operationalizing ML models with Cumulocity IoT and thin-edge.io for Visual quality Inspection [0.0]
  This paper introduces EdgeMLOps, a framework leveraging Cumu IoT and thin-edge.io for deploying and managing machine learning models on resource-constrained edge devices. We address the challenges of model optimization, deployment, and lifecycle management in edge environments. The framework's efficacy is demonstrated through a visual quality inspection (VQI) use case where images of assets are processed on edge devices, enabling real-time condition updates within an asset management system.
  arXiv  Detail & Related papers  (2025-01-28T16:40:40Z)
• Optimizing Small Language Models for In-Vehicle Function-Calling [4.148443557388842]
  We propose a holistic approach for deploying Small Language Models (SLMs) as function-calling agents within vehicles as edge devices. By leveraging SLMs, we simplify vehicle control mechanisms and enhance the user experience.
  arXiv  Detail & Related papers  (2025-01-04T17:32:56Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Development of an Edge Resilient ML Ensemble to Tolerate ICS Adversarial Attacks [0.9437165725355702]
  We build a resilient edge machine learning architecture that is designed to withstand adversarial attacks. The reML is based on the Resilient DDDAS paradigm, Moving Target Defense (MTD) theory, and TinyML. The proposed approach is power-efficient and privacy-preserving and, therefore, can be deployed on power-constrained devices to enhance ICS security.
  arXiv  Detail & Related papers  (2024-09-26T19:37:37Z)
• NeuPAN: Direct Point Robot Navigation with End-to-End Model-based Learning [67.53972459080437]
  Navigating a nonholonomic robot in a cluttered, unknown environment requires accurate perception and precise motion for real-time collision avoidance. This paper presents NeuPAN: a real-time, highly accurate, map-free, easy-to-deploy, and environment-invariant robot motion planner.
  arXiv  Detail & Related papers  (2024-03-11T15:44:38Z)
• Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms [59.03426963238452]
  We present Real-time Neural MPC, a framework to efficiently integrate large, complex neural network architectures as dynamics models within a model-predictive control pipeline. We show the feasibility of our framework on real-world problems by reducing the positional tracking error by up to 82% when compared to state-of-the-art MPC approaches without neural network dynamics.
  arXiv  Detail & Related papers  (2022-03-15T09:38:15Z)
• Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization [63.75188254377202]
  Deep reinforcement learning algorithms can perform poorly in real-world tasks due to discrepancy between source and target environments. We propose a novel model-free actor-critic algorithm to learn robust policies without modeling the disturbance in advance. Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.
  arXiv  Detail & Related papers  (2021-12-20T13:13:05Z)
• Neural Dynamic Policies for End-to-End Sensorimotor Learning [51.24542903398335]
  The current dominant paradigm in sensorimotor control, whether imitation or reinforcement learning, is to train policies directly in raw action spaces. We propose Neural Dynamic Policies (NDPs) that make predictions in trajectory distribution space. NDPs outperform the prior state-of-the-art in terms of either efficiency or performance across several robotic control tasks.
  arXiv  Detail & Related papers  (2020-12-04T18:59:32Z)
• Indoor Point-to-Point Navigation with Deep Reinforcement Learning and Ultra-wideband [1.6799377888527687]
  Moving obstacles and non-line-of-sight occurrences can generate noisy and unreliable signals. We show how a power-efficient point-to-point local planner, learnt with deep reinforcement learning (RL), can constitute a robust and resilient to noise short-range guidance system complete solution. Our results show that the computational efficient end-to-end policy learnt in plain simulation, can provide a robust, scalable and at-the-edge low-cost navigation system solution.
  arXiv  Detail & Related papers  (2020-11-18T12:30:36Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)
• Deep active inference agents using Monte-Carlo methods [3.8233569758620054]
  We present a neural architecture for building deep active inference agents in continuous state-spaces using Monte-Carlo sampling. Our approach enables agents to learn environmental dynamics efficiently, while maintaining task performance. Results show that deep active inference provides a flexible framework to develop biologically-inspired intelligent agents.
  arXiv  Detail & Related papers  (2020-06-07T15:10:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.