DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classification

• URL: http://arxiv.org/abs/2510.23203v1
• Date: Mon, 27 Oct 2025 10:46:22 GMT
• Title: DecoDINO: 3D Human-Scene Contact Prediction with Semantic Classification
• Abstract summary: We introduce DecoDINO, a three-branch network based on DECO's framework.<n>It uses two DINOv2 ViT-g/14 encoders, class-balanced loss weighting to reduce bias, and patch-level cross-attention for improved local reasoning.<n>On the DAMON benchmark, DecoDINO raises the binary-contact F1 score by 7$%$, (ii) halves the geodesic error, and (iii) augments predictions with object-level semantic labels.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate vertex-level contact prediction between humans and surrounding objects is a prerequisite for high fidelity human object interaction models used in robotics, AR/VR, and behavioral simulation. DECO was the first in the wild estimator for this task but is limited to binary contact maps and struggles with soft surfaces, occlusions, children, and false-positive foot contacts. We address these issues and introduce DecoDINO, a three-branch network based on DECO's framework. It uses two DINOv2 ViT-g/14 encoders, class-balanced loss weighting to reduce bias, and patch-level cross-attention for improved local reasoning. Vertex features are finally passed through a lightweight MLP with a softmax to assign semantic contact labels. We also tested a vision-language model (VLM) to integrate text features, but the simpler architecture performed better and was used instead. On the DAMON benchmark, DecoDINO (i) raises the binary-contact F1 score by 7$\%$, (ii) halves the geodesic error, and (iii) augments predictions with object-level semantic labels. Ablation studies show that LoRA fine-tuning and the dual encoders are key to these improvements. DecoDINO outperformed the challenge baseline in both tasks of the DAMON Challenge. Our code is available at https://github.com/DavidePasero/deco/tree/main.

Related papers

• Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation [59.968931594801234]
  Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging.<n>First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips.<n>Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose.
  arXiv  Detail & Related papers  (2026-06-29T17:38:12Z)
• PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models [23.263895549689863]
  Existing methods rely solely on next-token prediction loss, using only language tokens for supervision.<n>mname explicitly supervises intermediate point cloud tokens to preserve fine-grained 3D geometric-semantic information.<n>Experiments on ModelNet40 and averse datasets demonstrate that our method achieves textbf2.08 pp improvement on average for classification tasks.
  arXiv  Detail & Related papers  (2026-02-28T02:17:46Z)
• Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regression [12.590536117486257]
  Existing Vision Language Models (VLMs) struggle to comprehend real-world 3D spatial intelligence.<n> GEODE augments main VLM with two specialized, plug-and-play modules.<n>The synergy of these modules allows our 1.5B parameter model to function as a high-level semantic dispatcher.
  arXiv  Detail & Related papers  (2025-11-14T12:42:07Z)
• Detect Anything via Next Point Prediction [51.55967987350882]
  Rex- Omni is a 3B-scale MLLM that achieves state-of-the-art object perception performance.<n>On benchmarks like COCO and LVIS, Rex- Omni attains performance comparable to or exceeding regression-based models.
  arXiv  Detail & Related papers  (2025-10-14T17:59:54Z)
• TARGO: Benchmarking Target-driven Object Grasping under Occlusions [39.970680093124145]
  We first establish a new benchmark dataset for TARget-driven Grasping under Occlusions, named TARGO. We evaluate five grasp models and found that even the current SOTA model suffers when the occlusion level increases. We propose a transformer-based grasping model involving a shape completion module, termed TARGO-Net.
  arXiv  Detail & Related papers  (2024-07-08T17:47:45Z)
• Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion [57.232688209606515]
  We present HTCL, a novel Temporal Temporal Context Learning paradigm for improving camera-based semantic scene completion. Our method ranks $1st$ on the Semantic KITTI benchmark and even surpasses LiDAR-based methods in terms of mIoU.
  arXiv  Detail & Related papers  (2024-07-02T09:11:17Z)
• VoxelKP: A Voxel-based Network Architecture for Human Keypoint Estimation in LiDAR Data [53.638818890966036]
  textitVoxelKP is a novel fully sparse network architecture tailored for human keypoint estimation in LiDAR data. We introduce sparse box-attention to focus on learning spatial correlations between keypoints within each human instance. We incorporate a spatial encoding to leverage absolute 3D coordinates when projecting 3D voxels to a 2D grid encoding a bird's eye view.
  arXiv  Detail & Related papers  (2023-12-11T23:50:14Z)
• Enhancing Cross-Dataset Performance of Distracted Driving Detection With Score Softmax Classifier And Dynamic Gaussian Smoothing Supervision [6.891556476231427]
  Deep neural networks enable real-time monitoring of in-vehicle drivers, facilitating the timely prediction of distractions, fatigue, and potential hazards.<n>Recent research has exposed unreliable cross-dataset driver behavior recognition due to a limited number of data samples and background noise.<n>We propose a Score-Softmax classifier, which reduces the model overconfidence by enhancing category independence.
  arXiv  Detail & Related papers  (2023-10-08T15:28:01Z)
• Noisy-Correspondence Learning for Text-to-Image Person Re-identification [50.07634676709067]
  We propose a novel Robust Dual Embedding method (RDE) to learn robust visual-semantic associations even with noisy correspondences. Our method achieves state-of-the-art results both with and without synthetic noisy correspondences on three datasets.
  arXiv  Detail & Related papers  (2023-08-19T05:34:13Z)
• Improving Dual-Encoder Training through Dynamic Indexes for Negative Mining [61.09807522366773]
  We introduce an algorithm that approximates the softmax with provable bounds and that dynamically maintains the tree. In our study on datasets with over twenty million targets, our approach cuts error by half in relation to oracle brute-force negative mining.
  arXiv  Detail & Related papers  (2023-03-27T15:18:32Z)
• Self-supervised Human Mesh Recovery with Cross-Representation Alignment [20.69546341109787]
  Self-supervised human mesh recovery methods have poor generalizability due to limited availability and diversity of 3D-annotated benchmark datasets. We propose cross-representation alignment utilizing the complementary information from the robust but sparse representation (2D keypoints) This adaptive cross-representation alignment explicitly learns from the deviations and captures complementary information: richness from sparse representation and robustness from dense representation.
  arXiv  Detail & Related papers  (2022-09-10T04:47:20Z)
• Improving Top-K Decoding for Non-Autoregressive Semantic Parsing via Intent Conditioning [11.307865386100993]
  We propose a novel NAR semantic that introduces intent conditioning on the decoder. As the top-level intent governs the syntax and semantics of a parse, the intent conditioning allows the model to better control beam search. We evaluate the proposed NAR on conversational SP datasets, TOP & TOPv2.
  arXiv  Detail & Related papers  (2022-04-14T04:06:39Z)
• Progressive Coordinate Transforms for Monocular 3D Object Detection [52.00071336733109]
  We propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations. In this paper, we propose a novel and lightweight approach, dubbed em Progressive Coordinate Transforms (PCT) to facilitate learning coordinate representations.
  arXiv  Detail & Related papers  (2021-08-12T15:22:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.