Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics

• URL: http://arxiv.org/abs/2510.05558v1
• Date: Tue, 07 Oct 2025 04:07:44 GMT
• Title: Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics
• Authors: Christopher Hoang, Mengye Ren,
• Abstract summary: We present Midway Network, a new self-supervised learning architecture that is the first to learn strong visual representations for both object recognition and motion understanding solely from natural videos.<n>We show that Midway Network achieves strong performance on both semantic segmentation and optical flow tasks relative to prior self-supervised learning methods.
• Score: 11.610274121239051
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object recognition and motion understanding are key components of perception that complement each other. While self-supervised learning methods have shown promise in their ability to learn from unlabeled data, they have primarily focused on obtaining rich representations for either recognition or motion rather than both in tandem. On the other hand, latent dynamics modeling has been used in decision making to learn latent representations of observations and their transformations over time for control and planning tasks. In this work, we present Midway Network, a new self-supervised learning architecture that is the first to learn strong visual representations for both object recognition and motion understanding solely from natural videos, by extending latent dynamics modeling to this domain. Midway Network leverages a midway top-down path to infer motion latents between video frames, as well as a dense forward prediction objective and hierarchical structure to tackle the complex, multi-object scenes of natural videos. We demonstrate that after pretraining on two large-scale natural video datasets, Midway Network achieves strong performance on both semantic segmentation and optical flow tasks relative to prior self-supervised learning methods. We also show that Midway Network's learned dynamics can capture high-level correspondence via a novel analysis method based on forward feature perturbation.

Related papers

• Self-Supervised Video Representation Learning with Motion-Contrastive Perception [13.860736711747284]
  Motion-Contrastive Perception Network (MCPNet) MCPNet consists of two branches, namely, Motion Information Perception (MIP) and Contrastive Instance Perception (CIP) Our method outperforms current state-of-the-art visual-only self-supervised approaches.
  arXiv  Detail & Related papers  (2022-04-10T05:34:46Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)
• Self-supervised Video Object Segmentation by Motion Grouping [79.13206959575228]
  We develop a computer vision system able to segment objects by exploiting motion cues. We introduce a simple variant of the Transformer to segment optical flow frames into primary objects and the background. We evaluate the proposed architecture on public benchmarks (DAVIS2016, SegTrackv2, and FBMS59)
  arXiv  Detail & Related papers  (2021-04-15T17:59:32Z)
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery [84.73764603474413]
  We propose a novel online approach of multi-modal graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-11-03T11:00:10Z)
• Learning Long-term Visual Dynamics with Region Proposal Interaction Networks [75.06423516419862]
  We build object representations that can capture inter-object and object-environment interactions over a long-range. Thanks to the simple yet effective object representation, our approach outperforms prior methods by a significant margin.
  arXiv  Detail & Related papers  (2020-08-05T17:48:00Z)
• Hierarchical Contrastive Motion Learning for Video Action Recognition [100.9807616796383]
  We present hierarchical contrastive motion learning, a new self-supervised learning framework to extract effective motion representations from raw video frames. Our approach progressively learns a hierarchy of motion features that correspond to different abstraction levels in a network. Our motion learning module is lightweight and flexible to be embedded into various backbone networks.
  arXiv  Detail & Related papers  (2020-07-20T17:59:22Z)
• Distilled Semantics for Comprehensive Scene Understanding from Videos [53.49501208503774]
  In this paper, we take an additional step toward holistic scene understanding with monocular cameras by learning depth and motion alongside with semantics. We address the three tasks jointly by a novel training protocol based on knowledge distillation and self-supervision. We show that it yields state-of-the-art results for monocular depth estimation, optical flow and motion segmentation.
  arXiv  Detail & Related papers  (2020-03-31T08:52:13Z)
• Self-Supervised Joint Encoding of Motion and Appearance for First Person Action Recognition [19.93779132095822]
  We argue that learning features jointly intertwine from these two information channels is beneficial. We propose a single stream architecture able to do so, thanks to the addition of a self-supervised motion prediction block. Experiments on several publicly available databases show the power of our approach.
  arXiv  Detail & Related papers  (2020-02-10T17:51:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.