SpectralZoom: Efficient Segmentation with an Adaptive Hyperspectral Camera
- URL: http://arxiv.org/abs/2406.04287v1
- Date: Thu, 6 Jun 2024 17:33:23 GMT
- Title: SpectralZoom: Efficient Segmentation with an Adaptive Hyperspectral Camera
- Authors: Jackson Arnold, Sophia Rossi, Chloe Petrosino, Ethan Mitchell, Sanjeev J. Koppal,
- Abstract summary: We propose a vision transformer-based (ViT) algorithm that alleviates both the captured data footprint and the computational load for hyperspectral segmentation.
Our camera is able to adaptively sample image regions or patches at different resolutions, instead of capturing the entire hyperspectral cube at one high resolution.
- Score: 3.0175628677371935
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Hyperspectral image segmentation is crucial for many fields such as agriculture, remote sensing, biomedical imaging, battlefield sensing and astronomy. However, the challenge of hyper and multi spectral imaging is its large data footprint. We propose both a novel camera design and a vision transformer-based (ViT) algorithm that alleviate both the captured data footprint and the computational load for hyperspectral segmentation. Our camera is able to adaptively sample image regions or patches at different resolutions, instead of capturing the entire hyperspectral cube at one high resolution. Our segmentation algorithm works in concert with the camera, applying ViT-based segmentation only to adaptively selected patches. We show results both in simulation and on a real hardware platform demonstrating both accurate segmentation results and reduced computational burden.
Related papers
- SaccadeDet: A Novel Dual-Stage Architecture for Rapid and Accurate Detection in Gigapixel Images [50.742420049839474]
'SaccadeDet' is an innovative architecture for gigapixel-level object detection, inspired by the human eye saccadic movement.
Our approach, evaluated on the PANDA dataset, achieves an 8x speed increase over the state-of-the-art methods.
It also demonstrates significant potential in gigapixel-level pathology analysis through its application to Whole Slide Imaging.
arXiv Detail & Related papers (2024-07-25T11:22:54Z) - View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields [52.08335264414515]
We learn a novel feature field within a Neural Radiance Field (NeRF) representing a 3D scene.
Our method takes view-inconsistent multi-granularity 2D segmentations as input and produces a hierarchy of 3D-consistent segmentations as output.
We evaluate our method and several baselines on synthetic datasets with multi-view images and multi-granular segmentation, showcasing improved accuracy and viewpoint-consistency.
arXiv Detail & Related papers (2024-05-30T04:14:58Z) - Memory-Constrained Semantic Segmentation for Ultra-High Resolution UAV
Imagery [35.96063342025938]
This paper explores the intricate problem of achieving efficient and effective segmentation of ultra-high resolution UAV imagery.
We propose a GPU memory-efficient and effective framework for local inference without accessing the context beyond local patches.
We present an efficient memory-based interaction scheme to correct potential semantic bias of the underlying high-resolution information.
arXiv Detail & Related papers (2023-10-07T07:44:59Z) - Unsupervised Segmentation of Hyperspectral Remote Sensing Images with
Superpixels [22.92045376407794]
We propose an unsupervised method for hyperspectral remote sensing image segmentation.
The method exploits the mean-shift clustering algorithm that takes as input a preliminary hyperspectral superpixels segmentation together with the spectral pixel information.
Results demonstrate the validity of the proposed method in comparison with the state of the art.
arXiv Detail & Related papers (2022-04-26T13:20:33Z) - Real-time Hyperspectral Imaging in Hardware via Trained Metasurface
Encoders [55.16861072631285]
Hyperspectral imaging has attracted significant attention to identify spectral signatures for image classification and automated pattern recognition in computer vision.
This work introduces Hyplex, a new integrated architecture addressing the limitations discussed above.
Hyplex is a CMOS-compatible, fast hyperspectral camera that replaces bulk optics with nanoscale metasurfaces inversely designed through artificial intelligence.
arXiv Detail & Related papers (2022-04-05T09:52:51Z) - Panoptic SwiftNet: Pyramidal Fusion for Real-time Panoptic Segmentation [0.0]
Many applications require fast inference over large input resolutions on affordable or even embedded hardware.
We propose to achieve this goal by trading off backbone capacity for multi-scale feature extraction.
We present panoptic experiments on Cityscapes, Vistas, COCO and the BSB-Aerial dataset.
arXiv Detail & Related papers (2022-03-15T13:47:40Z) - Toward Efficient Hyperspectral Image Processing inside Camera Pixels [1.6449390849183356]
Hyperspectral cameras generate a large amount of data due to the presence of hundreds of spectral bands.
To mitigate this problem, we propose a form of processing-in-pixel (PIP)
Our PIP-optimized custom CNN layers effectively compress the input data, significantly reducing the bandwidth required to transmit the data downstream to the HSI processing unit.
arXiv Detail & Related papers (2022-03-11T01:06:02Z) - XCiT: Cross-Covariance Image Transformers [73.33400159139708]
We propose a "transposed" version of self-attention that operates across feature channels rather than tokens.
The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images.
arXiv Detail & Related papers (2021-06-17T17:33:35Z) - SASSI -- Super-Pixelated Adaptive Spatio-Spectral Imaging [17.2152544145501]
We introduce a novel video-rate hyperspectral imager with high spatial, and temporal resolutions.
A scene-adaptive spatial sampling of an hyperspectral scene, guided by its super-pixel segmented image, is capable of obtaining high-quality reconstructions.
We validate the proposed technique with extensive simulations as well as a lab prototype that measures hyperspectral video at a spatial resolution of $600 times 900$ pixels.
arXiv Detail & Related papers (2020-12-28T21:34:18Z) - Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral
Imagery [79.69449412334188]
In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches.
We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data.
Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
arXiv Detail & Related papers (2020-05-18T14:25:50Z) - Efficient Video Semantic Segmentation with Labels Propagation and
Refinement [138.55845680523908]
This paper tackles the problem of real-time semantic segmentation of high definition videos using a hybrid GPU / CPU approach.
We propose an Efficient Video(EVS) pipeline that combines: (i) On the CPU, a very fast optical flow method, that is used to exploit the temporal aspect of the video and propagate semantic information from one frame to the next.
On the popular Cityscapes dataset with high resolution frames (2048 x 1024), the proposed operating points range from 80 to 1000 Hz on a single GPU and CPU.
arXiv Detail & Related papers (2019-12-26T11:45:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.