STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding

• URL: http://arxiv.org/abs/2604.11637v1
• Date: Mon, 13 Apr 2026 15:47:23 GMT
• Title: STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding
• Abstract summary: We develop a framework that mixes spatial, temporal, and spectral representations of point cloud videos.<n> STS-Mixer achieves superior performance consistently across multiple widely adopted benchmarks on both 3D action recognition and 4D semantic segmentation tasks.
• Score: 88.18619962455948
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 4D point cloud videos capture rich spatial and temporal dynamics of scenes which possess unique values in various 4D understanding tasks. However, most existing methods work in the spatiotemporal domain where the underlying geometric characteristics of 4D point cloud videos are hard to capture, leading to degraded representation learning and understanding of 4D point cloud videos. We address the above challenge from a complementary spectral perspective. By transforming 4D point cloud videos into graph spectral signals, we can decompose them into multiple frequency bands each of which captures distinct geometric structures of point cloud videos. Our spectral analysis reveals that the decomposed low-frequency signals capture more coarse shapes while high-frequency signals encode more fine-grained geometry details. Building on these observations, we design Spatio-Temporal-Spectral Mixer (STS-Mixer), a unified framework that mixes spatial, temporal, and spectral representations of point cloud videos. STS-Mixer integrates multi-band delineated spectral signals with spatiotemporal information to capture rich geometries and temporal dynamics, while enabling fine-grained and holistic understanding of 4D point cloud videos. Extensive experiments show that STS-Mixer achieves superior performance consistently across multiple widely adopted benchmarks on both 3D action recognition and 4D semantic segmentation tasks. Code and models are available at https://github.com/Vegetebird/STS-Mixer.

Related papers

• Velox: Learning Representations of 4D Geometry and Appearance [37.12487786007913]
  We introduce a framework for learning latent representations of 4D objects which are descriptive, faithfully capturing object geometry and appearance.<n>We evaluate it across three downstream tasks -- video-to-4D generation, 3D tracking, and cloth simulation via image-to-4D generation.
  arXiv  Detail & Related papers  (2026-05-06T06:12:19Z)
• 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere [77.83037497484366]
  We present 4RC, a unified feed-forward framework for 4D reconstruction from monocular videos.<n>4RC learns a holistic 4D representation that jointly captures dense scene geometry and motion dynamics.
  arXiv  Detail & Related papers  (2026-02-10T18:57:04Z)
• Tracking-Guided 4D Generation: Foundation-Tracker Motion Priors for 3D Model Animation [21.075786141331974]
  We present emphTrack4DGen, a framework for generating dynamic 4D objects from sparse inputs.<n>In Stage One, we enforce dense, feature-level point correspondences inside the diffusion generator.<n>In Stage Two, we reconstruct a dynamic 4D-GS using a hybrid motion encoding.
  arXiv  Detail & Related papers  (2025-12-05T21:13:04Z)
• SyncTrack4D: Cross-Video Motion Alignment and Video Synchronization for Multi-Video 4D Gaussian Splatting [50.69165364520998]
  We present a novel multi-video 4D Gaussian Splatting (4DGS) approach designed to handle real-world, unsynchronized video sets.<n>Our approach, SyncTrack4D, directly leverages dense 4D track representation of dynamic scene parts as cues for simultaneous cross-video synchronization and 4DGS reconstruction.<n>We evaluate our approach on the Panoptic Studio and SyncNeRF Blender, demonstrating sub-frame synchronization accuracy with an average temporal error below 0.26 frames, and high-fidelity 4D reconstruction reaching 26.3 PSNR scores.
  arXiv  Detail & Related papers  (2025-12-03T23:05:01Z)
• Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models [79.06910348413861]
  We introduce Diff4Splat, a feed-forward method that synthesizes controllable and explicit 4D scenes from a single image.<n>Given a single input image, a camera trajectory, and an optional text prompt, Diff4Splat directly predicts a deformable 3D Gaussian field that encodes appearance, geometry, and motion.
  arXiv  Detail & Related papers  (2025-11-01T11:16:25Z)
• UST-SSM: Unified Spatio-Temporal State Space Models for Point Cloud Video Modeling [53.199942923818206]
  Point cloud videos capture 3D motion while reducing the effects of lighting and viewpoint variations, making them highly effective for recognizing subtle and continuous human actions.<n> Selective State Space Models (SSMs) have shown good performance in sequence modeling with linear complexity.<n>We propose the Unified Spatio-Temporal State Space Model (UST-SSM), which extends the latest advancements in SSMs to point cloud videos.
  arXiv  Detail & Related papers  (2025-08-20T10:46:01Z)
• MAMBA4D: Efficient Long-Sequence Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models [14.024240637175216]
  We propose a novel point cloud video understanding backbone based on the State Space Models (SSMs)<n> Specifically, we first disentangle space and time in 4D video sequences and then establish the spatial-temporal correlation with our designed Mamba blocks.<n>Our method has a significant efficiency improvement with 87.5% GPU memory reduction and 5.36 times speed-up.
  arXiv  Detail & Related papers  (2024-05-23T09:08:09Z)
• X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-modal Knowledge Transfer [28.719098240737605]
  We propose a novel cross-modal knowledge transfer framework, called X4D-SceneFormer. It enhances 4D-Scene understanding by transferring texture priors from RGB sequences using a Transformer architecture with temporal relationship mining. Experiments demonstrate the superior performance of our framework on various 4D point cloud video understanding tasks.
  arXiv  Detail & Related papers  (2023-12-12T15:48:12Z)
• PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition [55.38462937452363]
  We propose a unified multi-view cross-modal distillation architecture, including a pretrained deep image encoder as the teacher and a deep point encoder as the student. By pair-wise aligning multi-view visual and geometric descriptors, we can obtain more powerful deep point encoders without exhausting and complicated network modification.
  arXiv  Detail & Related papers  (2022-07-07T07:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.