Real-time event simulation with frame-based cameras

• URL: http://arxiv.org/abs/2209.04634v2
• Date: Thu, 23 Mar 2023 17:31:25 GMT
• Title: Real-time event simulation with frame-based cameras
• Authors: Andreas Ziegler, Daniel Teigland, Jonas Tebbe, Thomas Gossard and Andreas Zell
• Abstract summary: Event simulators minimize the need for real event cameras to develop novel algorithms. This work proposes simulation methods that improve the performance of event simulation by two orders of magnitude.
• Score: 13.045658279006524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Event cameras are becoming increasingly popular in robotics and computer vision due to their beneficial properties, e.g., high temporal resolution, high bandwidth, almost no motion blur, and low power consumption. However, these cameras remain expensive and scarce in the market, making them inaccessible to the majority. Using event simulators minimizes the need for real event cameras to develop novel algorithms. However, due to the computational complexity of the simulation, the event streams of existing simulators cannot be generated in real-time but rather have to be pre-calculated from existing video sequences or pre-rendered and then simulated from a virtual 3D scene. Although these offline generated event streams can be used as training data for learning tasks, all response time dependent applications cannot benefit from these simulators yet, as they still require an actual event camera. This work proposes simulation methods that improve the performance of event simulation by two orders of magnitude (making them real-time capable) while remaining competitive in the quality assessment.

Related papers

• Event Quality Score (EQS): Assessing the Realism of Simulated Event Camera Streams via Distances in Latent Space [20.537672896807063]
  Event cameras promise a paradigm shift in vision sensing with their low latency, high dynamic range, and asynchronous nature of events. We introduce event quality score (EQS), a quality metric that utilizes activations of the RVT architecture.
  arXiv  Detail & Related papers  (2025-04-16T22:25:57Z)
• Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation [62.5805866419814]
  Vid2Sim is a novel framework that bridges the sim2real gap through a scalable and cost-efficient real2sim pipeline for neural 3D scene reconstruction and simulation. Experiments demonstrate that Vid2Sim significantly improves the performance of urban navigation in the digital twins and real world by 31.2% and 68.3% in success rate.
  arXiv  Detail & Related papers  (2025-01-12T03:01:15Z)
• From Dashcam Videos to Driving Simulations: Stress Testing Automated Vehicles against Rare Events [5.132984904858975]
  Testing Automated Driving Systems (ADS) in simulation with realistic driving scenarios is important for verifying their performance. We propose a novel framework that automates the conversion of real-world car crash videos into detailed simulation scenarios. Our preliminary results demonstrate substantial time efficiency, finishing the real-to-sim conversion in minutes with full automation and no human intervention.
  arXiv  Detail & Related papers  (2024-11-25T01:01:54Z)
• ADV2E: Bridging the Gap Between Analogue Circuit and Discrete Frames in the Video-to-Events Simulator [6.783044920569469]
  Event cameras operate fundamentally differently from traditional Active Pixel Sensor (APS) cameras, offering significant advantages. Recent research has developed simulators to convert video frames into events, addressing the shortage of real event datasets. We propose a novel method of generating reliable event data based on a detailed analysis of the pixel circuitry in event cameras.
  arXiv  Detail & Related papers  (2024-11-19T05:52:51Z)
• Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions [56.84882059011291]
  We propose Deblur e-NeRF, a novel method to reconstruct blur-minimal NeRFs from motion-red events. We also introduce a novel threshold-normalized total variation loss to improve the regularization of large textureless patches.
  arXiv  Detail & Related papers  (2024-09-26T15:57:20Z)
• Scaling Face Interaction Graph Networks to Real World Scenes [12.519862235430153]
  We introduce a method which substantially reduces the memory required to run graph-based learned simulators. We show that our method uses substantially less memory than previous graph-based simulators while retaining their accuracy. This paves the way for expanding the application of learned simulators to settings where only perceptual information is available at inference time.
  arXiv  Detail & Related papers  (2024-01-22T14:38:25Z)
• Learning Interactive Real-World Simulators [96.5991333400566]
  We explore the possibility of learning a universal simulator of real-world interaction through generative modeling. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies. Video captioning models can benefit from training with simulated experience, opening up even wider applications.
  arXiv  Detail & Related papers  (2023-10-09T19:42:22Z)
• EvDNeRF: Reconstructing Event Data with Dynamic Neural Radiance Fields [80.94515892378053]
  EvDNeRF is a pipeline for generating event data and training an event-based dynamic NeRF. NeRFs offer geometric-based learnable rendering, but prior work with events has only considered reconstruction of static scenes. We show that by training on varied batch sizes of events, we can improve test-time predictions of events at fine time resolutions.
  arXiv  Detail & Related papers  (2023-10-03T21:08:41Z)
• Continual learning autoencoder training for a particle-in-cell simulation via streaming [52.77024349608834]
  upcoming exascale era will provide a new generation of physics simulations with high resolution. These simulations will have a high resolution, which will impact the training of machine learning models since storing a high amount of simulation data on disk is nearly impossible. This work presents an approach that trains a neural network concurrently to a running simulation without data on a disk.
  arXiv  Detail & Related papers  (2022-11-09T09:55:14Z)
• TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation [78.99283105497489]
  Event camera is a new device to enable video at the presence of arbitrarily complex motion. This paper proposes a novel TimeReplayer algorithm to interpolate videos captured by commodity cameras with events.
  arXiv  Detail & Related papers  (2022-03-25T18:57:42Z)
• Event Camera Simulator Design for Modeling Attention-based Inference Architectures [4.409836695738517]
  This paper presents an event camera simulator that can be a potent tool for hardware design prototyping. The proposed simulator implements a distributed computation model to identify relevant regions in an image frame. Our experimental results show that the simulator can effectively emulate event vision with low overheads.
  arXiv  Detail & Related papers  (2021-05-03T22:41:45Z)
• SimAug: Learning Robust Representations from Simulation for Trajectory Prediction [78.91518036949918]
  We propose a novel approach to learn robust representation through augmenting the simulation training data. We show that SimAug achieves promising results on three real-world benchmarks using zero real training data.
  arXiv  Detail & Related papers  (2020-04-04T21:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.