Event fields: Capturing light fields at high speed, resolution, and dynamic range

• URL: http://arxiv.org/abs/2412.06191v1
• Date: Mon, 09 Dec 2024 04:02:49 GMT
• Title: Event fields: Capturing light fields at high speed, resolution, and dynamic range
• Authors: Ziyuan Qu, Zihao Zou, Vivek Boominathan, Praneeth Chakravarthula, Adithya Pediredla,
• Abstract summary: "Event Fields" is a new approach that utilizes innovative optical designs for event cameras to capture light fields at high speed. We develop the underlying mathematical framework for Event Fields and introduce two foundational frameworks to capture them practically. This novel light-sensing paradigm opens doors to new applications in photography, robotics, and AR/VR, and presents fresh challenges in rendering and machine learning.
• Score: 9.2152453085337
• License:
• Abstract: Event cameras, which feature pixels that independently respond to changes in brightness, are becoming increasingly popular in high-speed applications due to their lower latency, reduced bandwidth requirements, and enhanced dynamic range compared to traditional frame-based cameras. Numerous imaging and vision techniques have leveraged event cameras for high-speed scene understanding by capturing high-framerate, high-dynamic range videos, primarily utilizing the temporal advantages inherent to event cameras. Additionally, imaging and vision techniques have utilized the light field-a complementary dimension to temporal information-for enhanced scene understanding. In this work, we propose "Event Fields", a new approach that utilizes innovative optical designs for event cameras to capture light fields at high speed. We develop the underlying mathematical framework for Event Fields and introduce two foundational frameworks to capture them practically: spatial multiplexing to capture temporal derivatives and temporal multiplexing to capture angular derivatives. To realize these, we design two complementary optical setups one using a kaleidoscope for spatial multiplexing and another using a galvanometer for temporal multiplexing. We evaluate the performance of both designs using a custom-built simulator and real hardware prototypes, showcasing their distinct benefits. Our event fields unlock the full advantages of typical light fields-like post-capture refocusing and depth estimation-now supercharged for high-speed and high-dynamic range scenes. This novel light-sensing paradigm opens doors to new applications in photography, robotics, and AR/VR, and presents fresh challenges in rendering and machine learning.

Related papers

• EventSplat: 3D Gaussian Splatting from Moving Event Cameras for Real-time Rendering [7.392798832833857]
  Event cameras offer exceptional temporal resolution and a high dynamic range. We introduce a method for using event camera data in novel view synthesis via Gaussian Splatting.
  arXiv  Detail & Related papers  (2024-12-10T08:23:58Z)
• Towards Real-world Event-guided Low-light Video Enhancement and Deblurring [39.942568142125126]
  Event cameras have emerged as a promising solution for improving image quality in low-light environments. We introduce an end-to-end framework to effectively handle these tasks. Our framework incorporates a module to efficiently leverage temporal information from events and frames.
  arXiv  Detail & Related papers  (2024-08-27T09:44:54Z)
• Radiance Fields from Photons [16.150757184515037]
  We introduce quanta radiance fields, a class of neural radiance fields that are trained at the granularity of individual photons using single-photon cameras (SPCs) We demonstrate, both via simulations and a prototype SPC hardware, high-fidelity reconstructions under high-speed motion, in low light, and for extreme dynamic range settings.
  arXiv  Detail & Related papers  (2024-07-12T16:06:51Z)
• Event-Based Motion Magnification [28.057537257958963]
  We propose a dual-camera system consisting of an event camera and a conventional RGB camera for video motion magnification. This innovative combination enables a broad and cost-effective amplification of high-frequency motions. We demonstrate the effectiveness and accuracy of our dual-camera system and network, offering a cost-effective and flexible solution for motion detection and magnification.
  arXiv  Detail & Related papers  (2024-02-19T08:59:58Z)
• E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning [53.63364311738552]
  Bio-inspired event cameras or dynamic vision sensors are capable of capturing per-pixel brightness changes (called event-streams) in high temporal resolution and high dynamic range. It calls for events-to-video (E2V) solutions which take event-streams as input and generate high quality video frames for intuitive visualization. We propose textbfE2HQV, a novel E2V paradigm designed to produce high-quality video frames from events.
  arXiv  Detail & Related papers  (2024-01-16T05:10:50Z)
• EventAid: Benchmarking Event-aided Image/Video Enhancement Algorithms with Real-captured Hybrid Dataset [55.12137324648253]
  Event cameras are emerging imaging technology that offers advantages over conventional frame-based imaging sensors in dynamic range and sensing speed. This paper focuses on five event-aided image and video enhancement tasks.
  arXiv  Detail & Related papers  (2023-12-13T15:42:04Z)
• Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion [67.15935067326662]
  Event cameras offer low power, low latency, high temporal resolution and high dynamic range. NeRF is seen as the leading candidate for efficient and effective scene representation. We propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras.
  arXiv  Detail & Related papers  (2023-09-15T17:52:08Z)
• Progressively Optimized Local Radiance Fields for Robust View Synthesis [76.55036080270347]
  We present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window.
  arXiv  Detail & Related papers  (2023-03-24T04:03:55Z)
• Fast Dynamic Radiance Fields with Time-Aware Neural Voxels [106.69049089979433]
  We propose a radiance field framework by representing scenes with time-aware voxel features, named as TiNeuVox. Our framework accelerates the optimization of dynamic radiance fields while maintaining high rendering quality. Our TiNeuVox completes training with only 8 minutes and 8-MB storage cost while showing similar or even better rendering performance than previous dynamic NeRF methods.
  arXiv  Detail & Related papers  (2022-05-30T17:47:31Z)
• Real-Time Optical Flow for Vehicular Perception with Low- and High-Resolution Event Cameras [3.845877724862319]
  Event cameras capture changes of illumination in the observed scene rather than accumulating light to create images. We propose an optimized framework for computing optical flow in real-time with both low- and high-resolution event cameras. We evaluate our approach on both low- and high-resolution driving sequences, and show that it often achieves better results than the current state of the art.
  arXiv  Detail & Related papers  (2021-12-20T15:09:20Z)
• ESL: Event-based Structured Light [62.77144631509817]
  Event cameras are bio-inspired sensors providing significant advantages over standard cameras. We propose a novel structured-light system using an event camera to tackle the problem of accurate and high-speed depth sensing.
  arXiv  Detail & Related papers  (2021-11-30T15:47:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.