Policy-based Foveated Imaging and Perception

• URL: http://arxiv.org/abs/2606.02565v1
• Date: Mon, 01 Jun 2026 17:55:05 GMT
• Title: Policy-based Foveated Imaging and Perception
• Abstract summary: We introduce a real-time, predictive, and task-aware foveated imaging system that operates directly at image acquisition time.<n>We formulate foveated acquisition as a sensor attention policy-learning problem, in which past observations guide actions that determine future measurements.<n>We validate our system on a 200-megapixel dual-stream sensor, capturing real-world videos under realistic bandwidth and latency constraints.
• Score: 36.66638469283988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ultra-high-resolution image sensors offer the potential to capture fine spatial details critical for many visual perception tasks, but acquiring and processing all pixels at full resolution is often infeasible under realistic bandwidth, latency, and power constraints. Existing approaches address this challenge through acquisition strategies such as spatial or temporal downsampling, which irrevocably discard information before task relevance can be assessed. In this work, we introduce a real-time, predictive, and task-aware foveated imaging system that operates directly at image acquisition time. Leveraging emerging dual-stream sensor architectures, our method dynamically allocates limited pixel bandwidth to task-relevant regions of interest while maintaining a low-resolution global context. We formulate foveated acquisition as a sensor attention policy-learning problem, in which past observations guide actions that determine future measurements, closing the perception-acquisition loop. Through extensive simulation across multiple perception tasks, we demonstrate that our approach achieves high task performance under strict pixel budgets and significantly outperforms relevant baselines operating at the same bandwidth. We further validate our system on a 200-megapixel dual-stream sensor, capturing real-world videos under realistic bandwidth and latency constraints, demonstrating the practical feasibility of task-driven, acquisition-time foveated imaging.

Related papers

• Event-based SLAM Benchmark for High-Speed Maneuvers [68.148886127117]
  Event-based cameras are bio-inspired sensors with pixels that independently and asynchronously respond to brightness changes at microsecond resolution.<n>Existing event-based approaches, although successful in mitigating motion blur caused by high-speed maneuvers, suffer from many limitations.<n>We introduce a benchmarking framework for event-based state estimation, called EvSLAM, characterized by sufficient variation in data collection platforms.
  arXiv  Detail & Related papers  (2026-04-27T04:36:41Z)
• Computational Imaging for Enhanced Computer Vision [0.0]
  This paper presents a comprehensive survey of computational imaging (CI) techniques and their transformative impact on computer vision (CV) applications.<n> Conventional imaging methods fail to deliver high-fidelity visual data in challenging conditions, such as low light, motion blur, or high dynamic range.<n> Computational imaging techniques, including light field imaging, high dynamic range (blurring) imaging, deblurring, high-speed imaging, and glare mitigation, address these limitations.
  arXiv  Detail & Related papers  (2025-09-10T16:02:42Z)
• SaccadeDet: A Novel Dual-Stage Architecture for Rapid and Accurate Detection in Gigapixel Images [50.742420049839474]
  'SaccadeDet' is an innovative architecture for gigapixel-level object detection, inspired by the human eye saccadic movement. Our approach, evaluated on the PANDA dataset, achieves an 8x speed increase over the state-of-the-art methods. It also demonstrates significant potential in gigapixel-level pathology analysis through its application to Whole Slide Imaging.
  arXiv  Detail & Related papers  (2024-07-25T11:22:54Z)
• Research, Applications and Prospects of Event-Based Pedestrian Detection: A Survey [10.494414329120909]
  Event-based cameras, inspired by the biological retina, have evolved into cutting-edge sensors distinguished by their minimal power requirements, negligible latency, superior temporal resolution, and expansive dynamic range. Event-based cameras address limitations by eschewing extraneous data transmissions and obviating motion blur in high-speed imaging scenarios. This paper offers an exhaustive review of research and applications particularly in the autonomous driving context.
  arXiv  Detail & Related papers  (2024-07-05T06:17:00Z)
• Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling [70.34875558830241]
  We present a way for learning a-temporal (4D) embedding, based on semantic semantic gears to allow for stratified modeling of dynamic regions of rendering the scene. At the same time, almost for free, our tracking approach enables free-viewpoint of interest - a functionality not yet achieved by existing NeRF-based methods.
  arXiv  Detail & Related papers  (2024-06-06T03:37:39Z)
• Resource Efficient Perception for Vision Systems [0.0]
  Our study introduces a framework aimed at mitigating these challenges by leveraging memory efficient patch based processing for high resolution images. It incorporates a global context representation alongside local patch information, enabling a comprehensive understanding of the image content. We demonstrate the effectiveness of our method through superior performance on 7 different benchmarks across classification, object detection, and segmentation.
  arXiv  Detail & Related papers  (2024-05-12T05:33:00Z)
• Generalizing Event-Based Motion Deblurring in Real-World Scenarios [62.995994797897424]
  Event-based motion deblurring has shown promising results by exploiting low-latency events. We propose a scale-aware network that allows flexible input spatial scales and enables learning from different temporal scales of motion blur. A two-stage self-supervised learning scheme is then developed to fit real-world data distribution.
  arXiv  Detail & Related papers  (2023-08-11T04:27:29Z)
• Efficient Human Vision Inspired Action Recognition using Adaptive Spatiotemporal Sampling [13.427887784558168]
  We introduce a novel adaptive vision system for efficient action recognition processing. Our system pre-scans the global context sampling scheme at low-resolution and decides to skip or request high-resolution features at salient regions for further processing. We validate the system on EPIC-KENS and UCF-101 datasets for action recognition, and show that our proposed approach can greatly speed up inference with a tolerable loss of accuracy compared with those from state-the-art baselines.
  arXiv  Detail & Related papers  (2022-07-12T01:18:58Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• Real-time single image depth perception in the wild with handheld devices [45.26484111468387]
  Two main issues limit depth estimation from handheld devices in-the-wild. We show how they are both addressable adopting appropriate network design and training strategies. We report experimental results concerning real-time depth-aware augmented reality and image blurring with smartphones in-the-wild.
  arXiv  Detail & Related papers  (2020-06-10T08:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.