Rewis3d: Reconstruction Improves Weakly-Supervised Semantic Segmentation

• URL: http://arxiv.org/abs/2603.06374v1
• Date: Fri, 06 Mar 2026 15:25:20 GMT
• Title: Rewis3d: Reconstruction Improves Weakly-Supervised Semantic Segmentation
• Abstract summary: Rewis3d is a framework that leverages recent advances in feed-forward 3D reconstruction to significantly improve weakly supervised semantic segmentation on 2D images.<n>We show that Rewis3d achieves state-of-the-art performance in sparse supervision, outperforming existing approaches by 2-7% without requiring additional labels or inference overhead.
• Score: 58.37682525044409
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Rewis3d, a framework that leverages recent advances in feed-forward 3D reconstruction to significantly improve weakly supervised semantic segmentation on 2D images. Obtaining dense, pixel-level annotations remains a costly bottleneck for training segmentation models. Alleviating this issue, sparse annotations offer an efficient weakly-supervised alternative. However, they still incur a performance gap. To address this, we introduce a novel approach that leverages 3D scene reconstruction as an auxiliary supervisory signal. Our key insight is that 3D geometric structure recovered from 2D videos provides strong cues that can propagate sparse annotations across entire scenes. Specifically, a dual student-teacher architecture enforces semantic consistency between 2D images and reconstructed 3D point clouds, using state-of-the-art feed-forward reconstruction to generate reliable geometric supervision. Extensive experiments demonstrate that Rewis3d achieves state-of-the-art performance in sparse supervision, outperforming existing approaches by 2-7% without requiring additional labels or inference overhead.

Related papers

• Pano3D: Unified 3D Reconstruction and Panoptic Segmentation [68.18993232814061]
  We introduce an approach that performs 3D reconstruction and 3D panoptic segmentation in a unified framework.<n>We build on existing 3D reconstruction models and augment them with a set-based mask decoder.<n>Our method achieves state-of-the-art performance in 3D panoptic segmentation across ScanNet, ScanNet200, and ScanNet++ datasets.
  arXiv  Detail & Related papers  (2026-06-12T09:43:13Z)
• RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations [70.83499963694238]
  RnG (Reconstruction and Generation) is a novel feed-forward Transformer that unifies reconstruction and generation.<n>It reconstructs visible geometry and generates plausible, coherent unseen geometry and appearance.<n>Our method achieves state-of-the-art performance in both generalizable 3D reconstruction and novel view generation.
  arXiv  Detail & Related papers  (2026-03-01T17:25:32Z)
• OVSeg3R: Learn Open-vocabulary Instance Segmentation from 2D via 3D Reconstruction [11.617237358347777]
  OVSeg3R learns open-vocabulary 3D instance segmentation from well-studied 2D perception models with the aid of 3D reconstruction.<n>OVSeg3R directly adopts reconstructed scenes from 2D videos as input, avoiding costly manual adjustment.<n>OVSeg3R substantially narrows the performance gap between tail and head classes, leading to an overall improvement of +2.3 mAP on the ScanNet200 benchmark.
  arXiv  Detail & Related papers  (2025-09-28T00:41:22Z)
• Reg3D: Reconstructive Geometry Instruction Tuning for 3D Scene Understanding [6.7958985137291235]
  Reg3D is a novel Reconstructive Geometry Instruction Tuning framework that incorporates geometry-aware supervision directly into the training process.<n>Our key insight is that effective 3D understanding requires reconstructing underlying geometric structures rather than merely describing them.<n>Experiments on ScanQA, Scan2Cap, ScanRefer, and SQA3D demonstrate that Reg3D delivers substantial performance improvements.
  arXiv  Detail & Related papers  (2025-09-03T18:36:44Z)
• PE3R: Perception-Efficient 3D Reconstruction [54.730257992806116]
  Perception-Efficient 3D Reconstruction (PE3R) is a novel framework designed to enhance both accuracy and efficiency.<n>The framework achieves a minimum 9-fold speedup in 3D semantic field reconstruction, along with substantial gains in perception accuracy and reconstruction precision.
  arXiv  Detail & Related papers  (2025-03-10T16:29:10Z)
• T-3DGS: Removing Transient Objects for 3D Scene Reconstruction [83.05271859398779]
  Transient objects in video sequences can significantly degrade the quality of 3D scene reconstructions.<n>We propose T-3DGS, a novel framework that robustly filters out transient distractors during 3D reconstruction using Gaussian Splatting.
  arXiv  Detail & Related papers  (2024-11-29T07:45:24Z)
• MOSE: Monocular Semantic Reconstruction Using NeRF-Lifted Noisy Priors [11.118490283303407]
  We propose a neural field semantic reconstruction approach to lift inferred image-level noisy priors to 3D. Our method produces accurate semantics and geometry in both 3D and 2D space.
  arXiv  Detail & Related papers  (2024-09-21T05:12:13Z)
• GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision [49.839374549646884]
  This paper presents GEOcc, a Geometric-Enhanced Occupancy network tailored for vision-only surround-view perception.<n>Our approach achieves State-Of-The-Art performance on the Occ3D-nuScenes dataset with the least image resolution needed and the most weightless image backbone.
  arXiv  Detail & Related papers  (2024-05-17T07:31:20Z)
• Learning monocular 3D reconstruction of articulated categories from motion [39.811816510186475]
  Video self-supervision forces the consistency of consecutive 3D reconstructions by a motion-based cycle loss. We introduce an interpretable model of 3D template deformations that controls a 3D surface through the displacement of a small number of local, learnable handles. We obtain state-of-the-art reconstructions with diverse shapes, viewpoints and textures for multiple articulated object categories.
  arXiv  Detail & Related papers  (2021-03-30T13:50:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.