MatSpectNet: Material Segmentation Network with Domain-Aware and Physically-Constrained Hyperspectral Reconstruction

• URL: http://arxiv.org/abs/2307.11466v4
• Date: Thu, 17 Aug 2023 09:19:57 GMT
• Title: MatSpectNet: Material Segmentation Network with Domain-Aware and Physically-Constrained Hyperspectral Reconstruction
• Authors: Yuwen Heng, Yihong Wu, Jiawen Chen, Srinandan Dasmahapatra, Hansung Kim
• Abstract summary: MatSpectNet is a new model to segment materials with recovered hyperspectral images from RGB images. It exploits the principles of colour perception in modern cameras to constrain the reconstructed hyperspectral images. It attains a 1.60% increase in average pixel accuracy and a 3.42% improvement in mean class accuracy compared with the most recent publication.
• Score: 13.451692195639696
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Achieving accurate material segmentation for 3-channel RGB images is challenging due to the considerable variation in a material's appearance. Hyperspectral images, which are sets of spectral measurements sampled at multiple wavelengths, theoretically offer distinct information for material identification, as variations in intensity of electromagnetic radiation reflected by a surface depend on the material composition of a scene. However, existing hyperspectral datasets are impoverished regarding the number of images and material categories for the dense material segmentation task, and collecting and annotating hyperspectral images with a spectral camera is prohibitively expensive. To address this, we propose a new model, the MatSpectNet to segment materials with recovered hyperspectral images from RGB images. The network leverages the principles of colour perception in modern cameras to constrain the reconstructed hyperspectral images and employs the domain adaptation method to generalise the hyperspectral reconstruction capability from a spectral recovery dataset to material segmentation datasets. The reconstructed hyperspectral images are further filtered using learned response curves and enhanced with human perception. The performance of MatSpectNet is evaluated on the LMD dataset as well as the OpenSurfaces dataset. Our experiments demonstrate that MatSpectNet attains a 1.60% increase in average pixel accuracy and a 3.42% improvement in mean class accuracy compared with the most recent publication. The project code is attached to the supplementary material and will be published on GitHub.

Related papers

• Towards virtual painting recolouring using Vision Transformer on X-Ray Fluorescence datacubes [80.32085982862151]
  We define a pipeline to perform virtual painting recolouring using raw data of X-Ray Fluorescence (XRF) analysis on pictorial artworks. To circumvent the small dataset size, we generate a synthetic dataset, starting from a database of XRF spectra. We define a Deep Variational Embedding network to embed the XRF spectra into a lower dimensional, K-Means friendly, metric space.
  arXiv  Detail & Related papers  (2024-10-11T14:05:28Z)
• Hyperspectral Dataset and Deep Learning methods for Waste from Electric and Electronic Equipment Identification (WEEE) [0.0]
  We evaluate the performance of diverse deep learning architectures for hyperspectral image segmentation. Results show that incorporating spatial information alongside spectral data leads to improved segmentation results. We contribute to the field by cleaning and publicly releasing the Tecnalia WEEE Hyperspectral dataset.
  arXiv  Detail & Related papers  (2024-07-05T13:45:11Z)
• OpenMaterial: A Comprehensive Dataset of Complex Materials for 3D Reconstruction [54.706361479680055]
  We introduce the OpenMaterial dataset, comprising 1001 objects made of 295 distinct materials. OpenMaterial provides comprehensive annotations, including 3D shape, material type, camera pose, depth, and object mask. It stands as the first large-scale dataset enabling quantitative evaluations of existing algorithms on objects with diverse and challenging materials.
  arXiv  Detail & Related papers  (2024-06-13T07:46:17Z)
• RMAFF-PSN: A Residual Multi-Scale Attention Feature Fusion Photometric Stereo Network [37.759675702107586]
  Predicting accurate maps of objects from two-dimensional images in regions of complex structure spatial material variations is challenging. We propose a method of calibrated feature information from different resolution stages and scales of the image. This approach preserves more physical information, such as texture and geometry of the object in complex regions.
  arXiv  Detail & Related papers  (2024-04-11T14:05:37Z)
• Limitations of Data-Driven Spectral Reconstruction -- Optics-Aware Analysis and Mitigation [22.07699685165064]
  Recent efforts in data-driven spectral reconstruction aim at extracting spectral information from RGB images captured by cost-effective RGB cameras. We evaluate both the practical limitations with respect to current datasets and overfitting, as well as fundamental limitations with respect to the nature of the information encoded in the RGB images. We propose to exploit the combination of metameric data augmentation and optical lens aberrations to improve the encoding of the metameric information into the RGB image.
  arXiv  Detail & Related papers  (2024-01-08T11:46:45Z)
• SpectralGPT: Spectral Remote Sensing Foundation Model [60.023956954916414]
  A universal RS foundation model, named SpectralGPT, is purpose-built to handle spectral RS images using a novel 3D generative pretrained transformer (GPT) Compared to existing foundation models, SpectralGPT accommodates input images with varying sizes, resolutions, time series, and regions in a progressive training fashion, enabling full utilization of extensive RS big data. Our evaluation highlights significant performance improvements with pretrained SpectralGPT models, signifying substantial potential in advancing spectral RS big data applications within the field of geoscience.
  arXiv  Detail & Related papers  (2023-11-13T07:09:30Z)
• Object Detection in Hyperspectral Image via Unified Spectral-Spatial Feature Aggregation [55.9217962930169]
  We present S2ADet, an object detector that harnesses the rich spectral and spatial complementary information inherent in hyperspectral images. S2ADet surpasses existing state-of-the-art methods, achieving robust and reliable results.
  arXiv  Detail & Related papers  (2023-06-14T09:01:50Z)
• MS-PS: A Multi-Scale Network for Photometric Stereo With a New Comprehensive Training Dataset [0.0]
  Photometric stereo (PS) problem consists in reconstructing the 3D-surface of an object. We propose a multi-scale architecture for PS which, combined with a new dataset, yields state-of-the-art results.
  arXiv  Detail & Related papers  (2022-11-25T14:01:54Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)
• Angular Luminance for Material Segmentation [6.374538197161135]
  Moving cameras provide multiple intensity measurements per pixel, yet often semantic segmentation, material recognition, and object recognition do not utilize this information. We utilize per-pixel angular luminance distributions as a key feature in discriminating the material of the surface. For real-world materials there is significant intra-class variation that can be managed by building a angular luminance network (AngLNet)
  arXiv  Detail & Related papers  (2020-09-22T21:15:27Z)
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery [79.69449412334188]
  In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches. We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
  arXiv  Detail & Related papers  (2020-05-18T14:25:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.