StreamTinyNet: video streaming analysis with spatial-temporal TinyML

• URL: http://arxiv.org/abs/2407.17524v1
• Date: Mon, 22 Jul 2024 07:08:03 GMT
• Title: StreamTinyNet: video streaming analysis with spatial-temporal TinyML
• Abstract summary: Video Streaming Analysis (VSA) is one of the most interesting tasks of TinyML. We present StreamTinyNet, the first TinyML architecture to perform multiple-frame VSA.
• Score: 2.5739385355356714
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Tiny Machine Learning (TinyML) is a branch of Machine Learning (ML) that constitutes a bridge between the ML world and the embedded system ecosystem (i.e., Internet of Things devices, embedded devices, and edge computing units), enabling the execution of ML algorithms on devices constrained in terms of memory, computational capabilities, and power consumption. Video Streaming Analysis (VSA), one of the most interesting tasks of TinyML, consists in scanning a sequence of frames in a streaming manner, with the goal of identifying interesting patterns. Given the strict constraints of these tiny devices, all the current solutions rely on performing a frame-by-frame analysis, hence not exploiting the temporal component in the stream of data. In this paper, we present StreamTinyNet, the first TinyML architecture to perform multiple-frame VSA, enabling a variety of use cases that requires spatial-temporal analysis that were previously impossible to be carried out at a TinyML level. Experimental results on public-available datasets show the effectiveness and efficiency of the proposed solution. Finally, StreamTinyNet has been ported and tested on the Arduino Nicla Vision, showing the feasibility of what proposed.

Related papers

• On The Dynamic Ensemble Selection for TinyML-based Systems -- a Preliminary Study [0.9553819152637493]
  Recent progress in TinyML technologies triggers the need to address the challenge of balancing inference time and classification quality.<n>This study examines a DES-Clustering approach for a multi-class computer vision task within TinyML systems.<n> Experiments have shown that a larger pool of classifiers for dynamic selection improves classification accuracy, and thus leads to an increase in average inference time on the TinyML device.
  arXiv  Detail & Related papers  (2025-09-22T18:35:35Z)
• RPCANet++: Deep Interpretable Robust PCA for Sparse Object Segmentation [51.37553739930992]
  RPCANet++ is a sparse object segmentation framework that fuses the interpretability of RPCA with efficient deep architectures.<n>Our approach unfolds a relaxed RPCA model into a structured network comprising a Background Approximation Module (BAM), an Object Extraction Module (OEM) and an Image Restoration Module (IRM)<n>Experiments on diverse datasets demonstrate that RPCANet++ achieves state-of-the-art performance under various imaging scenarios.
  arXiv  Detail & Related papers  (2025-08-06T08:19:37Z)
• Robust Multi-View Learning via Representation Fusion of Sample-Level Attention and Alignment of Simulated Perturbation [61.64052577026623]
  Real-world multi-view datasets are often heterogeneous and imperfect.<n>We propose a novel robust MVL method (namely RML) with simultaneous representation fusion and alignment.<n>Our RML is self-supervised and can also be applied for downstream tasks as a regularization.
  arXiv  Detail & Related papers  (2025-03-06T07:01:08Z)
• Token-Efficient Long Video Understanding for Multimodal LLMs [101.70681093383365]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs. We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Fast Data Aware Neural Architecture Search via Supernet Accelerated Evaluation [0.43550340493919387]
  Tiny machine learning (TinyML) promises to revolutionize fields such as healthcare, environmental monitoring, and industrial maintenance. The complex optimizations required for successful TinyML deployment continue to impede its widespread adoption. We propose a new state-of-the-art Data Aware Neural Architecture Search technique and demonstrate its effectiveness on the novel TinyML VisionWake' dataset.
  arXiv  Detail & Related papers  (2025-02-18T09:51:03Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• TinySV: Speaker Verification in TinyML with On-device Learning [2.356162747014486]
  This paper introduces a new type of adaptive TinyML solution that can be used in tasks, such as the presented textitTiny Speaker Verification (TinySV) The proposed TinySV solution relies on a two-layer hierarchical TinyML solution comprising Keyword Spotting and Adaptive Speaker Verification module. We evaluate the effectiveness and efficiency of the proposed TinySV solution on a dataset collected expressly for the task and tested the proposed solution on a real-world IoT device.
  arXiv  Detail & Related papers  (2024-06-03T17:27:40Z)
• Follow Anything: Open-set detection, tracking, and following in real-time [89.83421771766682]
  We present a robotic system to detect, track, and follow any object in real-time. Our approach, dubbed follow anything'' (FAn), is an open-vocabulary and multimodal model. FAn can be deployed on a laptop with a lightweight (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second.
  arXiv  Detail & Related papers  (2023-08-10T17:57:06Z)
• Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction [126.34551436845133]
  CNNs and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL) We present a novel MTL model by combining both merits of deformable CNN and query-based Transformer with shared gating for multi-task learning of dense prediction.
  arXiv  Detail & Related papers  (2023-08-10T17:37:49Z)
• Closing the loop: Autonomous experiments enabled by machine-learning-based online data analysis in synchrotron beamline environments [80.49514665620008]
  Machine learning can be used to enhance research involving large or rapidly generated datasets. In this study, we describe the incorporation of ML into a closed-loop workflow for X-ray reflectometry (XRR) We present solutions that provide an elementary data analysis in real time during the experiment without introducing the additional software dependencies in the beamline control software environment.
  arXiv  Detail & Related papers  (2023-06-20T21:21:19Z)
• Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention [100.81495948184649]
  We present Perceiver-VL, a vision-and-language framework that efficiently handles high-dimensional multimodal inputs such as long videos and text. Our framework scales with linear complexity, in contrast to the quadratic complexity of self-attention used in many state-of-the-art transformer-based models.
  arXiv  Detail & Related papers  (2022-11-21T18:22:39Z)
• A review of TinyML [0.0]
  The TinyML concept for embedded machine learning attempts to push such diversity from usual high-end approaches to low-end applications. TinyML is a rapidly expanding interdisciplinary topic at the convergence of machine learning, software, and hardware. This paper explores how TinyML can benefit a few specific industrial fields, its obstacles, and its future scope.
  arXiv  Detail & Related papers  (2022-11-05T06:02:08Z)
• An Ultra-low Power TinyML System for Real-time Visual Processing at Edge [7.327401565768275]
  This brief presents an extremely tiny backbone to construct high efficiency CNN models for various visual tasks. A specially designed neural co-processor (NCP) is interconnected with MCU to build an ultra-low power TinyML system. The proposed TinyML system based on our model, NCP and instruction set yields considerable accuracy and achieves a record ultra-low power of 160mW.
  arXiv  Detail & Related papers  (2022-07-11T06:56:27Z)
• How to Manage Tiny Machine Learning at Scale: An Industrial Perspective [5.384059021764428]
  Tiny machine learning (TinyML) has gained widespread popularity where machine learning (ML) is democratized on ubiquitous microcontrollers. TinyML models have been developed with different structures and are often distributed without a clear understanding of their working principles. We propose a framework using Semantic Web technologies to enable the joint management of TinyML models and IoT devices at scale.
  arXiv  Detail & Related papers  (2022-02-18T10:36:11Z)
• FAMINet: Learning Real-time Semi-supervised Video Object Segmentation with Steepest Optimized Optical Flow [21.45623125216448]
  Semi-supervised video object segmentation (VOS) aims to segment a few moving objects in a video sequence, where these objects are specified by annotation of first frame. The optical flow has been considered in many existing semi-supervised VOS methods to improve the segmentation accuracy. A FAMINet, which consists of a feature extraction network (F), an appearance network (A), a motion network (M), and an integration network (I), is proposed in this study to address the abovementioned problem.
  arXiv  Detail & Related papers  (2021-11-20T07:24:33Z)
• Feature Flow: In-network Feature Flow Estimation for Video Object Detection [56.80974623192569]
  Optical flow is widely used in computer vision tasks to provide pixel-level motion information. A common approach is to:forward optical flow to a neural network and fine-tune this network on the task dataset. We propose a novel network (IFF-Net) with an textbfIn-network textbfFeature textbfFlow estimation module for video object detection.
  arXiv  Detail & Related papers  (2020-09-21T07:55:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.