Bidirectional Cross-Attention Fusion of High-Res RGB and Low-Res HSI for Multimodal Automated Waste Sorting

• URL: http://arxiv.org/abs/2603.13941v1
• Date: Sat, 14 Mar 2026 13:21:36 GMT
• Title: Bidirectional Cross-Attention Fusion of High-Res RGB and Low-Res HSI for Multimodal Automated Waste Sorting
• Abstract summary: We present Bi Cross-Attention Fusion (BCAF), which aligns high-resolution RGB with low-resolution HSI at their native grids via localized, bidirectional cross-attention.<n>BCAF achieves state-of-the-art performance of 76.4% mIoU at 31 images/s and 75.4% mIoU at 55 images/s.
• Score: 0.7166329363666318
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Growing waste streams and the transition to a circular economy require efficient automated waste sorting. In industrial settings, materials move on fast conveyor belts, where reliable identification and ejection demand pixel-accurate segmentation. RGB imaging delivers high-resolution spatial detail, which is essential for accurate segmentation, but it confuses materials that look similar in the visible spectrum. Hyperspectral imaging (HSI) provides spectral signatures that separate such materials, yet its lower spatial resolution limits detail. Effective waste sorting therefore needs methods that fuse both modalities to exploit their complementary strengths. We present Bidirectional Cross-Attention Fusion (BCAF), which aligns high-resolution RGB with low-resolution HSI at their native grids via localized, bidirectional cross-attention, avoiding pre-upsampling or early spectral collapse. BCAF uses two independent backbones: a standard Swin Transformer for RGB and an HSI-adapted Swin backbone that preserves spectral structure through 3D tokenization with spectral self-attention. We also analyze trade-offs between RGB input resolution and the number of HSI spectral slices. Although our evaluation targets RGB-HSI fusion, BCAF is modality-agnostic and applies to co-registered RGB with lower-resolution, high-channel auxiliary sensors. On the benchmark SpectralWaste dataset, BCAF achieves state-of-the-art performance of 76.4% mIoU at 31 images/s and 75.4% mIoU at 55 images/s. We further evaluate a novel industrial dataset: K3I-Cycling (first RGB subset already released on Fordatis). On this dataset, BCAF reaches 62.3% mIoU for material segmentation (paper, metal, plastic, etc.) and 66.2% mIoU for plastic-type segmentation (PET, PP, HDPE, LDPE, PS, etc.).

Related papers

• Lightweight Interpretable RGB-Guided Hyperspectral Super-Resolution under Real Cross-resolution Misalignment [6.5685272845309255]
  RGB-guided hyperspectral super-resolution transfers spatial detail from a high-resolution RGB guide to a low-resolution hyperspectral image.<n>We propose a lightweight and interpretable RGB-guided HSR framework combining cross-modal flow alignment with model-based Gram-Schmidtization fusion.
  arXiv  Detail & Related papers  (2026-09-01T10:55:11Z)
• HIR-ALIGN: Enhancing Hyperspectral Image Restoration via Diffusion-Based Data Generation [50.46910397782266]
  Hyperspectral image (HSI) restoration is crucial for reliable analysis, as real HSIs suffer from degradations like noise, blur, and resolution loss.<n>We present HIR-ALIGN, a plug-and-play target-adaptive augmentation framework that enhances hyperspectral image restoration by augmenting limited training images with synthetic data that closely matches the target distribution using no extra data.
  arXiv  Detail & Related papers  (2026-05-13T14:14:13Z)
• SegFly: A 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale [42.70356809606482]
  We propose a scalable geometry-driven 2D-3D-2D paradigm for semantic segmentation of aerial imagery.<n>By lifting less than 3% of RGB images into a semantic 3D point cloud, we produce 97% of RGB labels and 100% of thermal labels.<n>We extend this 2D-3D-2D paradigm to cross-modal image registration, using 3D geometry as an intermediate alignment space.
  arXiv  Detail & Related papers  (2026-03-18T16:57:22Z)
• Modality-Decoupled RGB-Thermal Object Detector via Query Fusion [15.717929078660227]
  We propose a Modality-Decoupled RGB-T detection framework with Query Fusion (MDQF) to balance modality complementation and separation.<n>Our approach delivers superior performance to existing RGB-T detectors and achieves better modality independence.
  arXiv  Detail & Related papers  (2026-01-13T11:32:29Z)
• CSNR and JMIM Based Spectral Band Selection for Reducing Metamerism in Urban Driving [3.2953155328671038]
  metamerism is a phenomenon where distinct materials appear similar in RGB imagery.<n>This work investigates hyperspectral imaging (HSI) to overcome this limitation by capturing unique material signatures beyond the visible spectrum.
  arXiv  Detail & Related papers  (2025-08-14T13:50:51Z)
• RGB Pre-Training Enhanced Unobservable Feature Latent Diffusion Model for Spectral Reconstruction [16.54284634377436]
  We propose a two-stage pipeline consisting of spectral structure representation learning and spectral-spatial joint distribution learning.<n>In the first stage, a spectral unobservable feature autoencoder (SpeUAE) is trained to extract and compress the unobservable feature into a 3D manifold aligned with RGB space.<n>The ULDM is then acquired to model the distribution of the coded unobservable feature with guidance from the corresponding RGB images.
  arXiv  Detail & Related papers  (2025-07-17T10:07:32Z)
• Bringing RGB and IR Together: Hierarchical Multi-Modal Enhancement for Robust Transmission Line Detection [67.02804741856512]
  We propose a novel Hierarchical Multi-Modal Enhancement Network (HMMEN) that integrates RGB and IR data for robust and accurate TL detection.<n>Our method introduces two key components: (1) a Mutual Multi-Modal Enhanced Block (MMEB), which fuses and enhances hierarchical RGB and IR feature maps in a coarse-to-fine manner, and (2) a Feature Alignment Block (FAB) that corrects misalignments between decoder outputs and IR feature maps by leveraging deformable convolutions.
  arXiv  Detail & Related papers  (2025-01-25T06:21:06Z)
• Ternary-Type Opacity and Hybrid Odometry for RGB NeRF-SLAM [58.736472371951955]
  We introduce a ternary-type opacity (TT) model, which categorizes points on a ray intersecting a surface into three regions: before, on, and behind the surface. This enables a more accurate rendering of depth, subsequently improving the performance of image warping techniques. Our integrated approach of TT and HO achieves state-of-the-art performance on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-12-20T18:03:17Z)
• Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution [52.582632746409665]
  We propose a novel Symmetric Uncertainty-aware Feature Transmission (SUFT) for color-guided DSR. Our method achieves superior performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-06-01T06:35:59Z)
• Robust RGB-D Fusion for Saliency Detection [13.705088021517568]
  We propose a robust RGB-D fusion method that benefits from layer-wise and trident spatial, attention mechanisms. Our experiments on five benchmark datasets demonstrate that the proposed fusion method performs consistently better than the state-of-the-art fusion alternatives.
  arXiv  Detail & Related papers  (2022-08-02T21:23:00Z)
• Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction [64.10636296274168]
  Hyperspectral pansharpening aims to synthesize a low-resolution hyperspectral image (LR-HSI) with a registered panchromatic image (PAN) to generate an enhanced HSI with high spectral and spatial resolution. Recently proposed HS pansharpening methods have obtained remarkable results using deep convolutional networks (ConvNets) We propose a novel over-complete network, called HyperKite, which focuses on learning high-level features by constraining the receptive from increasing in the deep layers.
  arXiv  Detail & Related papers  (2021-07-06T14:11:03Z)
• Tuning IR-cut Filter for Illumination-aware Spectral Reconstruction from RGB [84.1657998542458]
  It has been proven that the reconstruction accuracy relies heavily on the spectral response of the RGB camera in use. This paper explores the filter-array based color imaging mechanism of existing RGB cameras, and proposes to design the IR-cut filter properly for improved spectral recovery.
  arXiv  Detail & Related papers  (2021-03-26T19:42:21Z)
• Fast Hyperspectral Image Recovery via Non-iterative Fusion of Dual-Camera Compressive Hyperspectral Imaging [22.683482662362337]
  Coded aperture snapshot spectral imaging (CASSI) is a promising technique to capture the three-dimensional hyperspectral image (HSI) Various regularizers have been exploited to reconstruct the 3D data from the 2D measurement. One feasible solution is to utilize additional information such as the RGB measurement in CASSI.
  arXiv  Detail & Related papers  (2020-12-30T10:29:32Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.