Enhancing Self-Supervised Fine-Grained Video Object Tracking with Dynamic Memory Prediction

• URL: http://arxiv.org/abs/2504.21692v1
• Date: Wed, 30 Apr 2025 14:29:04 GMT
• Title: Enhancing Self-Supervised Fine-Grained Video Object Tracking with Dynamic Memory Prediction
• Authors: Zihan Zhou, Changrui Dai, Aibo Song, Xiaolin Fang,
• Abstract summary: We introduce a Dynamic Memory Prediction framework that utilizes multiple reference frames to concisely enhance frame reconstruction.<n>Our algorithm outperforms the state-of-the-art self-supervised techniques on two fine-grained video object tracking tasks.
• Score: 5.372301053935416
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Successful video analysis relies on accurate recognition of pixels across frames, and frame reconstruction methods based on video correspondence learning are popular due to their efficiency. Existing frame reconstruction methods, while efficient, neglect the value of direct involvement of multiple reference frames for reconstruction and decision-making aspects, especially in complex situations such as occlusion or fast movement. In this paper, we introduce a Dynamic Memory Prediction (DMP) framework that innovatively utilizes multiple reference frames to concisely and directly enhance frame reconstruction. Its core component is a Reference Frame Memory Engine that dynamically selects frames based on object pixel features to improve tracking accuracy. In addition, a Bidirectional Target Prediction Network is built to utilize multiple reference frames to improve the robustness of the model. Through experiments, our algorithm outperforms the state-of-the-art self-supervised techniques on two fine-grained video object tracking tasks: object segmentation and keypoint tracking.

Related papers

• Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling [7.3949576464066]
  We propose a deep learning framework designed to significantly optimize bandwidth for motion-transfer-enabled video applications. To capture complex motion effectively, we utilize the First Order Motion Model (FOMM), which encodes dynamic objects by detecting keypoints. We validate our results across three datasets for video animation and reconstruction using the following metrics: Mean Absolute Error, Joint Embedding Predictive Architecture Embedding Distance, Structural Similarity Index, and Average Pair-wise Displacement.
  arXiv  Detail & Related papers  (2025-04-07T22:21:54Z)
• Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better [61.381599921020175]
  Temporal consistency is critical in video prediction to ensure that outputs are coherent and free of artifacts.<n>Traditional methods, such as temporal attention and 3D convolution, may struggle with significant object motion.<n>We propose the Tracktention Layer, a novel architectural component that explicitly integrates motion information using point tracks.
  arXiv  Detail & Related papers  (2025-03-25T17:58:48Z)
• Scoring, Remember, and Reference: Catching Camouflaged Objects in Videos [24.03405963900272]
  Video Camouflaged Object Detection aims to segment objects whose appearances closely resemble their surroundings. Existing vision models often struggle in such scenarios due to the indistinguishable appearance of camouflaged objects. We propose an end-to-end framework inspired by human memory-recognition.
  arXiv  Detail & Related papers  (2025-03-21T11:08:14Z)
• Understanding Long Videos via LLM-Powered Entity Relation Graphs [51.13422967711056]
  GraphVideoAgent is a framework that maps and monitors the evolving relationships between visual entities throughout the video sequence.<n>Our approach demonstrates remarkable effectiveness when tested against industry benchmarks.
  arXiv  Detail & Related papers  (2025-01-27T10:57:24Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• Dyna-DepthFormer: Multi-frame Transformer for Self-Supervised Depth Estimation in Dynamic Scenes [19.810725397641406]
  We propose a novel Dyna-Depthformer framework, which predicts scene depth and 3D motion field jointly. Our contributions are two-fold. First, we leverage multi-view correlation through a series of self- and cross-attention layers in order to obtain enhanced depth feature representation. Second, we propose a warping-based Motion Network to estimate the motion field of dynamic objects without using semantic prior.
  arXiv  Detail & Related papers  (2023-01-14T09:43:23Z)
• Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation [49.17930380106643]
  We propose a novel framework to speed up any architecture with skip-connections for real-time vision tasks. Specifically, at the arrival of each frame, we transform the features from the previous frame to reuse them at specific spatial bins. We then perform partial computation of the backbone network on the regions of the current frame that captures temporal differences between the current and previous frame.
  arXiv  Detail & Related papers  (2022-06-20T07:20:02Z)
• Implicit Motion Handling for Video Camouflaged Object Detection [60.98467179649398]
  We propose a new video camouflaged object detection (VCOD) framework. It can exploit both short-term and long-term temporal consistency to detect camouflaged objects from video frames.
  arXiv  Detail & Related papers  (2022-03-14T17:55:41Z)
• Motion-aware Dynamic Graph Neural Network for Video Compressive Sensing [14.67994875448175]
  Video snapshot imaging (SCI) utilizes a 2D detector to capture sequential video frames and compress them into a single measurement. Most existing reconstruction methods are incapable of efficiently capturing long-range spatial and temporal dependencies. We propose a flexible and robust approach based on the graph neural network (GNN) to efficiently model non-local interactions between pixels in space and time regardless of the distance.
  arXiv  Detail & Related papers  (2022-03-01T12:13:46Z)
• Siamese Network with Interactive Transformer for Video Object Segmentation [34.202137199782804]
  We propose a network with a specifically designed interactive transformer, called SITVOS, to enable effective context propagation from historical to current frames. We employ the backbone architecture to extract backbone features of both past and current frames, which enables feature reuse and is more efficient than existing methods.
  arXiv  Detail & Related papers  (2021-12-28T03:38:17Z)
• ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring [92.40655035360729]
  Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions. We propose a novel implicit method to learn spatial correspondence among blurry frames in the feature space. Our proposed method is evaluated on the widely-adopted DVD dataset, along with a newly collected High-Frame-Rate (1000 fps) dataset for Video Deblurring.
  arXiv  Detail & Related papers  (2021-03-07T04:33:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.