Comparison of Spatiotemporal Networks for Learning Video Related Tasks

• URL: http://arxiv.org/abs/2009.07338v1
• Date: Tue, 15 Sep 2020 19:57:50 GMT
• Title: Comparison of Spatiotemporal Networks for Learning Video Related Tasks
• Authors: Logan Courtney, Ramavarapu Sreenivas
• Abstract summary: Many methods for learning from sequences involve temporally processing 2D CNN features from the individual frames or directly utilizing 3D convolutions within high-performing 2D CNN architectures. This work constructs an MNIST-based video dataset with parameters controlling relevant facets of common video-related tasks: classification, ordering, and speed estimation. Models trained on this dataset are shown to differ in key ways depending on the task and their use of 2D convolutions, 3D convolutions, or convolutional LSTMs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many methods for learning from video sequences involve temporally processing 2D CNN features from the individual frames or directly utilizing 3D convolutions within high-performing 2D CNN architectures. The focus typically remains on how to incorporate the temporal processing within an already stable spatial architecture. This work constructs an MNIST-based video dataset with parameters controlling relevant facets of common video-related tasks: classification, ordering, and speed estimation. Models trained on this dataset are shown to differ in key ways depending on the task and their use of 2D convolutions, 3D convolutions, or convolutional LSTMs. An empirical analysis indicates a complex, interdependent relationship between the spatial and temporal dimensions with design choices having a large impact on a network's ability to learn the appropriate spatiotemporal features.

Related papers

• Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion [57.232688209606515]
  We present HTCL, a novel Temporal Temporal Context Learning paradigm for improving camera-based semantic scene completion. Our method ranks $1st$ on the Semantic KITTI benchmark and even surpasses LiDAR-based methods in terms of mIoU.
  arXiv  Detail & Related papers  (2024-07-02T09:11:17Z)
• Dynamic 3D Point Cloud Sequences as 2D Videos [81.46246338686478]
  3D point cloud sequences serve as one of the most common and practical representation modalities of real-world environments. We propose a novel generic representation called textitStructured Point Cloud Videos (SPCVs) SPCVs re-organizes a point cloud sequence as a 2D video with spatial smoothness and temporal consistency, where the pixel values correspond to the 3D coordinates of points.
  arXiv  Detail & Related papers  (2024-03-02T08:18:57Z)
• Multi-task Learning with 3D-Aware Regularization [55.97507478913053]
  We propose a structured 3D-aware regularizer which interfaces multiple tasks through the projection of features extracted from an image encoder to a shared 3D feature space. We show that the proposed method is architecture agnostic and can be plugged into various prior multi-task backbones to improve their performance.
  arXiv  Detail & Related papers  (2023-10-02T08:49:56Z)
• SeMLaPS: Real-time Semantic Mapping with Latent Prior Networks and Quasi-Planar Segmentation [53.83313235792596]
  We present a new methodology for real-time semantic mapping from RGB-D sequences. It combines a 2D neural network and a 3D network based on a SLAM system with 3D occupancy mapping. Our system achieves state-of-the-art semantic mapping quality within 2D-3D networks-based systems.
  arXiv  Detail & Related papers  (2023-06-28T22:36:44Z)
• Spatio-Temporal Self-Attention Network for Video Saliency Prediction [13.873682190242365]
  3D convolutional neural networks have achieved promising results for video tasks in computer vision. We propose a novel Spatio-Temporal Self-Temporal Self-Attention 3 Network (STSANet) for video saliency prediction.
  arXiv  Detail & Related papers  (2021-08-24T12:52:47Z)
• Spatial-Temporal Correlation and Topology Learning for Person Re-Identification in Videos [78.45050529204701]
  We propose a novel framework to pursue discriminative and robust representation by modeling cross-scale spatial-temporal correlation. CTL utilizes a CNN backbone and a key-points estimator to extract semantic local features from human body. It explores a context-reinforced topology to construct multi-scale graphs by considering both global contextual information and physical connections of human body.
  arXiv  Detail & Related papers  (2021-04-15T14:32:12Z)
• Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks [87.50632573601283]
  We present a novel method for multi-view depth estimation from a single video. Our method achieves temporally coherent depth estimation results by using a novel Epipolar Spatio-Temporal (EST) transformer. To reduce the computational cost, inspired by recent Mixture-of-Experts models, we design a compact hybrid network.
  arXiv  Detail & Related papers  (2020-11-26T04:04:21Z)
• Making a Case for 3D Convolutions for Object Segmentation in Videos [16.167397418720483]
  We show that 3D convolutional networks can be effectively applied to dense video prediction tasks such as salient object segmentation. We propose a 3D decoder architecture, that comprises novel 3D Global Convolution layers and 3D Refinement modules. Our approach outperforms existing state-of-the-arts by a large margin on the DAVIS'16 Unsupervised, FBMS and ViSal benchmarks.
  arXiv  Detail & Related papers  (2020-08-26T12:24:23Z)
• STH: Spatio-Temporal Hybrid Convolution for Efficient Action Recognition [39.58542259261567]
  We present a novel S-Temporal Hybrid Network (STH) which simultaneously encodes spatial and temporal video information with a small parameter. Such a design enables efficient-temporal modeling and maintains a small model scale. STH enjoys performance superiority over 3D CNNs while maintaining an even smaller parameter cost than 2D CNNs.
  arXiv  Detail & Related papers  (2020-03-18T04:46:30Z)
• Interpreting video features: a comparison of 3D convolutional networks and convolutional LSTM networks [1.462434043267217]
  We compare how 3D convolutional networks and convolutional LSTM networks learn features across temporally dependent frames. Our findings indicate that the 3D convolutional model concentrates on shorter events in the input sequence, and places its spatial focus on fewer, contiguous areas.
  arXiv  Detail & Related papers  (2020-02-02T11:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.