SVL: Spike-based Vision-language Pretraining for Efficient 3D Open-world Understanding

• URL: http://arxiv.org/abs/2505.17674v1
• Date: Fri, 23 May 2025 09:41:10 GMT
• Title: SVL: Spike-based Vision-language Pretraining for Efficient 3D Open-world Understanding
• Abstract summary: Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D-temporal understanding.<n>Existing SNNs exhibit a significant performance gap compared to Artificial Neural Networks (ANNs)<n>We propose a Spike-based Vision-Language (SVL) pretraining framework that empowers SNNs with open-world 3D understanding.
• Score: 12.75012908465477
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D spatio-temporal features. However, existing SNNs still exhibit a significant performance gap compared to Artificial Neural Networks (ANNs) due to inadequate pre-training strategies. These limitations manifest as restricted generalization ability, task specificity, and a lack of multimodal understanding, particularly in challenging tasks such as multimodal question answering and zero-shot 3D classification. To overcome these challenges, we propose a Spike-based Vision-Language (SVL) pretraining framework that empowers SNNs with open-world 3D understanding while maintaining spike-driven efficiency. SVL introduces two key components: (i) Multi-scale Triple Alignment (MTA) for label-free triplet-based contrastive learning across 3D, image, and text modalities, and (ii) Re-parameterizable Vision-Language Integration (Rep-VLI) to enable lightweight inference without relying on large text encoders. Extensive experiments show that SVL achieves a top-1 accuracy of 85.4% in zero-shot 3D classification, surpassing advanced ANN models, and consistently outperforms prior SNNs on downstream tasks, including 3D classification (+6.1%), DVS action recognition (+2.1%), 3D detection (+1.1%), and 3D segmentation (+2.1%) with remarkable efficiency. Moreover, SVL enables SNNs to perform open-world 3D question answering, sometimes outperforming ANNs. To the best of our knowledge, SVL represents the first scalable, generalizable, and hardware-friendly paradigm for 3D open-world understanding, effectively bridging the gap between SNNs and ANNs in complex open-world understanding tasks. Code is available https://github.com/bollossom/SVL.

Related papers

• Exploring Bottlenecks in VLM-LLM Navigation: How 3D Scene Understanding Capability Impacts Zero-Shot VLN [6.626149978783011]
  Zero-shot vision-and-language navigation (VLN) has gained significant attention due to its minimal data collection costs and inherent generalizations.<n>Current 3D perception models prioritize pixel-level accuracy, directly conflicting with the strict computational limits and real-time efficiency demanded by embodied navigation.<n>This paper quantifies the impact of 3D scene understanding capability on VLN performance.
  arXiv  Detail & Related papers  (2026-05-14T13:12:05Z)
• Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMs [37.792545310147965]
  This paper presents Efficient3D, a unified framework for visual token pruning.<n>It achieves superior performance compared with unpruned baselines, with a +2.57% CIDEr improvement on the Scan2Cap dataset.
  arXiv  Detail & Related papers  (2026-04-03T03:32:55Z)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models [100.13033631690114]
  Vision-language models (VLMs) struggle with 3D-related tasks such as spatial cognition and physical understanding.<n>We introduce SandboxVLM, a framework that leverages abstract bounding boxes to encode geometric structure and physical kinematics for VLM.<n>Our approach consistently improves spatial intelligence, achieving an 8.3% gain on SAT Real compared with baseline methods.
  arXiv  Detail & Related papers  (2025-11-14T04:16:09Z)
• OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision [31.929268076595122]
  OccVLA is a novel framework that integrates 3D occupancy representations into a unified multimodal reasoning process.<n>OccVLA achieves state-of-the-art results on the nuScenes benchmark for trajectory planning and demonstrates superior performance on 3D visual question-answering tasks.
  arXiv  Detail & Related papers  (2025-09-06T03:47:21Z)
• Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding [24.964149224068027]
  We propose Fast3D, a plug-and-play visual token pruning framework for 3D MLLMs.<n>Global Attention Prediction (GAP) learns to predict the global attention distributions of the target model, enabling efficient token importance estimation.<n>SAP, introduces dynamic token budgets through attention-based complexity assessment, automatically adjusting layer-wise pruning ratios.
  arXiv  Detail & Related papers  (2025-07-12T16:29:02Z)
• econSG: Efficient and Multi-view Consistent Open-Vocabulary 3D Semantic Gaussians [56.85804719947]
  We propose econSG for open-vocabulary semantic segmentation with 3DGS.<n>Our econSG shows state-of-the-art performance on four benchmark datasets compared to the existing methods.
  arXiv  Detail & Related papers  (2025-04-08T13:12:31Z)
• Activation-wise Propagation: A Universal Strategy to Break Timestep Constraints in Spiking Neural Networks for 3D Data Processing [29.279985043923386]
  We introduce Activation-wise Membrane Potential Propagation (AMP2), a novel state update mechanism for spiking neurons.<n>Inspired by skip connections in deep networks, AMP2 incorporates the membrane potential of neurons into network, eliminating the need for iterative updates.<n>Our method achieves significant improvements across various 3D modalities, including 3D point clouds and event streams.
  arXiv  Detail & Related papers  (2025-02-18T11:52:25Z)
• Efficient 3D Recognition with Event-driven Spike Sparse Convolution [15.20476631850388]
  Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D-temporal features.<n>We introduce the Spike Voxel Coding (SVC) scheme, which encodes the 3D point clouds into a sparse spike train space.<n>We propose a Spike Sparse Convolution (SSC) model for efficiently extracting 3D sparse point cloud features.
  arXiv  Detail & Related papers  (2024-12-10T09:55:15Z)
• Generalized Robot 3D Vision-Language Model with Fast Rendering and Pre-Training Vision-Language Alignment [55.11291053011696]
  This work presents a framework for dealing with 3D scene understanding when the labeled scenes are quite limited.<n>To extract knowledge for novel categories from the pre-trained vision-language models, we propose a hierarchical feature-aligned pre-training and knowledge distillation strategy.<n>In the limited reconstruction case, our proposed approach, termed WS3D++, ranks 1st on the large-scale ScanNet benchmark.
  arXiv  Detail & Related papers  (2023-12-01T15:47:04Z)
• ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights [61.36309876889977]
  ViT-Lens enables efficient omni-modal representation learning by perceiving novel modalities with a pretrained ViT and aligning to a pre-defined space. In zero-shot 3D classification, ViT-Lens achieves substantial improvements over previous state-of-the-art. We will release the results of ViT-Lens on more modalities in the near future.
  arXiv  Detail & Related papers  (2023-08-20T07:26:51Z)
• Artificial-Spiking Hierarchical Networks for Vision-Language Representation Learning [16.902924543372713]
  State-of-the-art methods achieve impressive performance by pre-training on large-scale datasets. We propose an efficient framework for multimodal alignment by introducing a novel visual semantic module. Experiments show that the proposed ASH-Nets achieve competitive results.
  arXiv  Detail & Related papers  (2023-08-18T10:40:25Z)
• VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud [51.063494002003154]
  3D semantic scene graph (3DSSG) prediction in the point cloud is challenging since the 3D point cloud only captures geometric structures with limited semantics compared to 2D images. We propose Visual-Linguistic Semantics Assisted Training scheme that can significantly empower 3DSSG prediction models with discrimination about long-tailed and ambiguous semantic relations.
  arXiv  Detail & Related papers  (2023-03-25T09:14:18Z)
• ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding [110.07170245531464]
  Current 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. Recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. We learn a unified representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities.
  arXiv  Detail & Related papers  (2022-12-10T01:34:47Z)
• Learning from Unlabeled 3D Environments for Vision-and-Language Navigation [87.03299519917019]
  In vision-and-language navigation (VLN), an embodied agent is required to navigate in realistic 3D environments following natural language instructions. We propose to automatically create a large-scale VLN dataset from 900 unlabeled 3D buildings from HM3D. We experimentally demonstrate that HM3D-AutoVLN significantly increases the generalization ability of resulting VLN models.
  arXiv  Detail & Related papers  (2022-08-24T21:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.