TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos

• URL: http://arxiv.org/abs/2605.01234v1
• Date: Sat, 02 May 2026 04:14:40 GMT
• Title: TT4D: A Pipeline and Dataset for Table Tennis 4D Reconstruction From Monocular Videos
• Abstract summary: We present TT4D, a large-scale, high-fidelity table tennis dataset.<n>It provides $140+$ hours of reconstructed singles and doubles gameplay from monocular broadcast videos.<n>This rich data provides a new foundation for virtual replay, in-depth player analysis, and robot learning.
• Score: 20.74832764427194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present TT4D, a large-scale, high-fidelity table tennis dataset. It provides $140+$ hours of reconstructed singles and doubles gameplay from monocular broadcast videos, featuring multimodal annotations like high-quality camera calibrations, precise 3D ball positions, ball spin, time segmentation, and 3D human meshes over time. This rich data provides a new foundation for virtual replay, in-depth player analysis, and robot learning. The dataset's combination of scale and precision is achieved through a novel reconstruction pipeline. Prior methods first partition a game sequence into individual shot segments based on the 2D ball track, and only then attempt reconstruction. However, 2D-based time segmentation collapses under occlusion and varied camera viewpoints, preventing reliable reconstruction. We invert this paradigm by first lifting the entire unsegmented 2D ball track to 3D through a learned lifting network. This 3D trajectory then allows us to reliably perform time segmentation. The learned lifting network also infers the ball's spin, handles unreliable ball detections, and successfully reconstructs the ball trajectory in cases of high occlusion. This lift-first design is necessary, as our pipeline is the only method capable of reconstructing table tennis gameplay from general-view broadcast monocular videos. We demonstrate the dataset's fidelity through two downstream tasks: estimating the racket's pose \& velocity at impact, and training a generative model of competitive rallies.

Related papers

• Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Video [81.44600627066747]
  Mesh4D is a feed-forward model for monocular 4D mesh reconstruction.<n>Our key contribution is a compact latent space that encodes the entire animation sequence in a single pass.<n>We evaluate Mesh4D on reconstruction and view novel benchmarks, outperforming prior methods in recovering accurate 3D shape and deformation.
  arXiv  Detail & Related papers  (2026-01-08T18:59:56Z)
• Uplifting Table Tennis: A Robust, Real-World Application for 3D Trajectory and Spin Estimation [32.884975524869084]
  We propose a novel two-stage pipeline that divides the problem into a front-end perception task and a back-end 2D-to-3D uplifting task.<n>By integrating a ball detector and a table keypoint detector, our approach transforms a proof-of-concept uplifting method into a practical, robust, and high-performing end-to-end application.
  arXiv  Detail & Related papers  (2025-11-25T12:25:20Z)
• C4D: 4D Made from 3D through Dual Correspondences [77.04731692213663]
  We introduce C4D, a framework that leverages temporal correspondences to extend existing 3D reconstruction formulation to 4D.<n>C4D captures two types of correspondences: short-term optical flow and long-term point tracking.<n>We train a dynamic-aware point tracker that provides additional mobility information.
  arXiv  Detail & Related papers  (2025-10-16T17:59:06Z)
• SpatialTrackerV2: 3D Point Tracking Made Easy [73.0350898700048]
  SpatialTrackerV2 is a feed-forward 3D point tracking method for monocular videos.<n>It decomposes world-space 3D motion into scene geometry, camera ego-motion, and pixel-wise object motion.<n>By learning geometry and motion jointly from such heterogeneous data, SpatialTrackerV2 outperforms existing 3D tracking methods by 30%.
  arXiv  Detail & Related papers  (2025-07-16T17:59:03Z)
• Where Is The Ball: 3D Ball Trajectory Estimation From 2D Monocular Tracking [10.237629959021875]
  We present a method for 3D ball trajectory estimation from a 2D tracking sequence.<n>Our method achieves state-of-the-art performance despite training solely on simulated data.<n>Our method can generalize to real-world scenarios with multiple trajectories, opening up a range of applications in sport analysis and virtual replay.
  arXiv  Detail & Related papers  (2025-06-06T05:42:05Z)
• SoccerNet-v3D: Leveraging Sports Broadcast Replays for 3D Scene Understanding [16.278222277579655]
  We introduce SoccerNet-v3D and ISSIA-3D, two datasets designed for 3D scene understanding in soccer broadcast analysis.<n>These datasets extend SoccerNet-v3 and ISSIA by incorporating field-line-based camera calibration and multi-view synchronization.<n>We propose a monocular 3D ball localization task built upon the triangulation of ground-truth 2D ball annotations.
  arXiv  Detail & Related papers  (2025-04-14T11:15:13Z)
• TT3D: Table Tennis 3D Reconstruction [11.84899291358663]
  We propose a novel approach for reconstructing precise 3D ball trajectories from online table tennis match recordings.<n>Our method leverages the underlying physics of the ball's motion to identify the bounce state that minimizes the reprojection error of the ball's flying trajectory.<n>A key advantage of our approach is its ability to infer ball spin without relying on human pose estimation or racket tracking.
  arXiv  Detail & Related papers  (2025-04-14T09:37:47Z)
• VGGT: Visual Geometry Grounded Transformer [61.37669770946458]
  VGGT is a feed-forward neural network that directly infers all key 3D attributes of a scene.<n>Network achieves state-of-the-art results in multiple 3D tasks.
  arXiv  Detail & Related papers  (2025-03-14T17:59:47Z)
• Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos [76.07894127235058]
  We present a system for mining high-quality 4D reconstructions from internet stereoscopic, wide-angle videos.<n>We use this method to generate large-scale data in the form of world-consistent, pseudo-metric 3D point clouds.<n>We demonstrate the utility of this data by training a variant of DUSt3R to predict structure and 3D motion from real-world image pairs.
  arXiv  Detail & Related papers  (2024-12-12T18:59:54Z)
• Neural Network-Based Tracking and 3D Reconstruction of Baseball Pitch Trajectories from Single-View 2D Video [0.0]
  We present a neural network-based approach for tracking and reconstructing the trajectories of baseball pitches from 2D video footage to 3D coordinates. Our experimental results demonstrate that this approach achieves high accuracy in reconstructing 3D trajectories from 2D inputs.
  arXiv  Detail & Related papers  (2024-05-25T16:17:10Z)
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm [111.16358607889609]
  We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation.<n>For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.