Transtreaming: Adaptive Delay-aware Transformer for Real-time Streaming Perception

• URL: http://arxiv.org/abs/2409.06584v1
• Date: Tue, 10 Sep 2024 15:26:38 GMT
• Title: Transtreaming: Adaptive Delay-aware Transformer for Real-time Streaming Perception
• Abstract summary: This work presents an innovative real-time streaming perception method, Transtreaming, which addresses the challenge of real-time object detection with dynamic computational delay. The proposed model outperforms the existing state-of-the-art methods, even in single-frame detection scenarios. Transtreaming meets the stringent real-time processing requirements on all kinds of devices.
• Score: 18.403242474776764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time object detection is critical for the decision-making process for many real-world applications, such as collision avoidance and path planning in autonomous driving. This work presents an innovative real-time streaming perception method, Transtreaming, which addresses the challenge of real-time object detection with dynamic computational delay. The core innovation of Transtreaming lies in its adaptive delay-aware transformer, which can concurrently predict multiple future frames and select the output that best matches the real-world present time, compensating for any system-induced computation delays. The proposed model outperforms the existing state-of-the-art methods, even in single-frame detection scenarios, by leveraging a transformer-based methodology. It demonstrates robust performance across a range of devices, from powerful V100 to modest 2080Ti, achieving the highest level of perceptual accuracy on all platforms. Unlike most state-of-the-art methods that struggle to complete computation within a single frame on less powerful devices, Transtreaming meets the stringent real-time processing requirements on all kinds of devices. The experimental results emphasize the system's adaptability and its potential to significantly improve the safety and reliability for many real-world systems, such as autonomous driving.

Related papers

• StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models [84.17128030384099]
  StreamPI is a streaming multimodal temporal modeling framework.<n>It equips single-frame VLA with temporal reasoning capability without introducing any additional parameters.<n> Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks.
  arXiv  Detail & Related papers  (2026-08-26T17:33:19Z)
• MotionDLO: Hybrid Event- and Frame-Based Tracking of Deformable Linear Objects [0.0]
  We introduce MotionDLO, a real-time tracking framework specifically designed to overcome limitations in temporal continuity and latency.<n>The method exploits the high temporal resolution and sparsity of event-based cameras and combines segmentation with the Coherent Point Drift (CPD) algorithm.<n>We demonstrate that the proposed framework reliably associates DLO instances across video sequences, enabling robust perception for robotic manipulation tasks.
  arXiv  Detail & Related papers  (2026-08-23T12:53:13Z)
• On Exploring Input Resolution Scaling For Anytime LiDAR Object Detection [4.014524824655106]
  We focus on enabling anytime computing for deep neural networks (DNNs) that process LiDAR point clouds for 3D object detection.<n>We propose a novel method that enables multi-resolution inference for models that process point clouds as pillars or voxels.<n>Our memory-efficient approach requires the deployment of only a single DNN model, avoiding the need to deploy multiple models.
  arXiv  Detail & Related papers  (2026-07-09T12:16:23Z)
• Invertible Neural Network Adapter for One-Step Flow Matching in Robot Manipulation [48.77928097573051]
  This paper presents an invertible neural network adapter for general robotic manipulation.<n>Built upon a flow-matching formulation, the proposed adapter constrains the action generation trajectory within an invertible latent space.<n>Experiments are conducted across a diverse set of simulation benchmarks and real-world robotic platforms to evaluate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2026-06-17T15:35:27Z)
• Streaming Real-Time Trajectory Prediction Using Endpoint-Aware Modeling [54.94692733670454]
  Future trajectories of neighboring traffic agents have a significant influence on the path planning and decision-making of autonomous vehicles.<n>We propose a lightweight yet highly accurate streaming-based trajectory forecasting approach.<n>Our approach significantly reduces inference latency, making it well-suited for real-world deployment.
  arXiv  Detail & Related papers  (2026-03-02T13:44:23Z)
• Lightweight Temporal Transformer Decomposition for Federated Autonomous Driving [11.79541267274746]
  We propose a method that processes sequential image frames and temporal steering data by breaking down large attention maps into smaller matrices.<n>This approach reduces model complexity, enabling efficient weight updates for convergence and real-time predictions.<n>Experiments on three datasets demonstrate that our method outperforms recent approaches by a clear margin while achieving real-time performance.
  arXiv  Detail & Related papers  (2025-06-30T05:14:16Z)
• DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving [62.62464518137153]
  DriveTransformer is a simplified E2E-AD framework for the ease of scaling up. It is composed of three unified operations: task self-attention, sensor cross-attention, temporal cross-attention. It achieves state-of-the-art performance in both simulated closed-loop benchmark Bench2Drive and real world open-loop benchmark nuScenes with high FPS.
  arXiv  Detail & Related papers  (2025-03-07T11:41:18Z)
• Estimating Vehicle Speed on Roadways Using RNNs and Transformers: A Video-based Approach [0.0]
  This project explores the application of advanced machine learning models, specifically Long Short-Term Memory (LSTM), Gated Recurrent Units (GRU), and Transformers, to the task of vehicle speed estimation using video data.
  arXiv  Detail & Related papers  (2025-02-21T15:51:49Z)
• CorrDiff: Adaptive Delay-aware Detector with Temporal Cue Inputs for Real-time Object Detection [11.714072240331518]
  CorrDiff is designed to tackle the challenge of delays in real-time detection systems. It is able to utilize runtime-estimated temporal cues to predict objects' locations for multiple future frames. It meets the stringent real-time processing requirements on all kinds of devices.
  arXiv  Detail & Related papers  (2025-01-09T10:34:25Z)
• Event-Aided Time-to-Collision Estimation for Autonomous Driving [28.13397992839372]
  We present a novel method that estimates the time to collision using a neuromorphic event-based camera. The proposed algorithm consists of a two-step approach for efficient and accurate geometric model fitting on event data. Experiments on both synthetic and real data demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-07-10T02:37:36Z)
• Transformer-based Fusion of 2D-pose and Spatio-temporal Embeddings for Distracted Driver Action Recognition [8.841708075914353]
  Temporal localization of driving actions over time is important for advanced driver-assistance systems and naturalistic driving studies. We aim to improve the temporal localization and classification accuracy performance by adapting video action recognition and 2D human-based estimation networks to one model. The model performs well on the A2 test set the 2023 NVIDIA AI City Challenge for naturalistic driving action recognition.
  arXiv  Detail & Related papers  (2024-03-11T10:26:38Z)
• MTD: Multi-Timestep Detector for Delayed Streaming Perception [0.5439020425819]
  Streaming perception is a task of reporting the current state of the world, which is used to evaluate the delay and accuracy of autonomous driving systems. This paper propose the Multi- Timestep Detector (MTD), an end-to-end detector which uses dynamic routing for multi-branch future prediction. The proposed method has been evaluated on the Argoverse-HD dataset, and the experimental results show that it has achieved state-of-the-art performance across various delay settings.
  arXiv  Detail & Related papers  (2023-09-13T06:23:58Z)
• StreamYOLO: Real-time Object Detection for Streaming Perception [84.2559631820007]
  We endow the models with the capacity of predicting the future, significantly improving the results for streaming perception. We consider multiple velocities driving scene and propose Velocity-awared streaming AP (VsAP) to jointly evaluate the accuracy. Our simple method achieves the state-of-the-art performance on Argoverse-HD dataset and improves the sAP and VsAP by 4.7% and 8.2% respectively.
  arXiv  Detail & Related papers  (2022-07-21T12:03:02Z)
• Scalable Vehicle Re-Identification via Self-Supervision [66.2562538902156]
  Vehicle Re-Identification is one of the key elements in city-scale vehicle analytics systems. Many state-of-the-art solutions for vehicle re-id mostly focus on improving the accuracy on existing re-id benchmarks and often ignore computational complexity. We propose a simple yet effective hybrid solution empowered by self-supervised training which only uses a single network during inference time.
  arXiv  Detail & Related papers  (2022-05-16T12:14:42Z)
• Real-time Object Detection for Streaming Perception [84.2559631820007]
  Streaming perception is proposed to jointly evaluate the latency and accuracy into a single metric for video online perception. We build a simple and effective framework for streaming perception. Our method achieves competitive performance on Argoverse-HD dataset and improves the AP by 4.9% compared to the strong baseline.
  arXiv  Detail & Related papers  (2022-03-23T11:33:27Z)
• Adaptive Anomaly Detection for Internet of Things in Hierarchical Edge Computing: A Contextual-Bandit Approach [81.5261621619557]
  We propose an adaptive anomaly detection scheme with hierarchical edge computing (HEC) We first construct multiple anomaly detection DNN models with increasing complexity, and associate each of them to a corresponding HEC layer. Then, we design an adaptive model selection scheme that is formulated as a contextual-bandit problem and solved by using a reinforcement learning policy network.
  arXiv  Detail & Related papers  (2021-08-09T08:45:47Z)
• Real-time Streaming Perception System for Autonomous Driving [2.6058660721533187]
  We present the real-time steaming perception system, which is also the 2nd Place solution of Streaming Perception Challenge. Unlike traditional object detection challenges, which focus mainly on the absolute performance, streaming perception task requires achieving a balance of accuracy and latency. On the Argoverse-HD test set, our method achieves 33.2 streaming AP (34.6 streaming AP verified by the organizer) under the required hardware.
  arXiv  Detail & Related papers  (2021-07-30T01:32:44Z)
• Intrinsic Temporal Regularization for High-resolution Human Video Synthesis [59.54483950973432]
  temporal consistency is crucial for extending image processing pipelines to the video domain. We propose an effective intrinsic temporal regularization scheme, where an intrinsic confidence map is estimated via the frame generator to regulate motion estimation. We apply our intrinsic temporal regulation to single-image generator, leading to a powerful " INTERnet" capable of generating $512times512$ resolution human action videos.
  arXiv  Detail & Related papers  (2020-12-11T05:29:45Z)
• Towards Streaming Perception [70.68520310095155]
  We present an approach that coherently integrates latency and accuracy into a single metric for real-time online perception. The key insight behind this metric is to jointly evaluate the output of the entire perception stack at every time instant. We focus on the illustrative tasks of object detection and instance segmentation in urban video streams, and contribute a novel dataset with high-quality and temporally-dense annotations.
  arXiv  Detail & Related papers  (2020-05-21T01:51:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.