Unsupervised Fish Trajectory Tracking and Segmentation

• URL: http://arxiv.org/abs/2208.10662v1
• Date: Tue, 23 Aug 2022 01:01:27 GMT
• Title: Unsupervised Fish Trajectory Tracking and Segmentation
• Authors: Alzayat Saleh, Marcus Sheaves, Dean Jerry, Mostafa Rahimi Azghadi
• Abstract summary: We propose a three-stage framework for robust fish tracking and segmentation. The first stage is an optical flow model, which generates the pseudo labels using spatial and temporal consistency between frames. In the second stage, a self-supervised model refines the pseudo-labels incrementally. In the third stage, the refined labels are used to train a segmentation network.
• Score: 2.1028463367241033
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: DNN for fish tracking and segmentation based on high-quality labels is expensive. Alternative unsupervised approaches rely on spatial and temporal variations that naturally occur in video data to generate noisy pseudo-ground-truth labels. These pseudo-labels are used to train a multi-task deep neural network. In this paper, we propose a three-stage framework for robust fish tracking and segmentation, where the first stage is an optical flow model, which generates the pseudo labels using spatial and temporal consistency between frames. In the second stage, a self-supervised model refines the pseudo-labels incrementally. In the third stage, the refined labels are used to train a segmentation network. No human annotations are used during the training or inference. Extensive experiments are performed to validate our method on three public underwater video datasets and to demonstrate that it is highly effective for video annotation and segmentation. We also evaluate the robustness of our framework to different imaging conditions and discuss the limitations of our current implementation.

Related papers

• CBIL: Collective Behavior Imitation Learning for Fish from Real Videos [58.81930297206828]
  We present a scalable approach, Collective Behavior Imitation Learning (CBIL), for learning fish schooling behavior directly from videos. MVAE effectively maps 2D observations to implicit states that are compact and expressive for following the imitation learning stage. CBIL can be used for various animation tasks with the learned collective motion priors.
  arXiv  Detail & Related papers  (2025-03-31T21:15:00Z)
• Closer to Ground Truth: Realistic Shape and Appearance Labeled Data Generation for Unsupervised Underwater Image Segmentation [8.511846002129522]
  We introduce a novel two stage unsupervised segmentation approach that requires no human annotations and combines artificially created and real images. Our method generates challenging synthetic training data, by placing virtual fish in real-world underwater habitats. We show its effectiveness on the specific case of salmon segmentation in underwater videos, for which we introduce DeepSalmon, the largest dataset of its kind in the literature (30 GB)
  arXiv  Detail & Related papers  (2025-03-20T11:34:45Z)
• Image-Based Relocalization and Alignment for Long-Term Monitoring of Dynamic Underwater Environments [57.59857784298534]
  We propose an integrated pipeline that combines Visual Place Recognition (VPR), feature matching, and image segmentation on video-derived images. This method enables robust identification of revisited areas, estimation of rigid transformations, and downstream analysis of ecosystem changes.
  arXiv  Detail & Related papers  (2025-03-06T05:13:19Z)
• TrajSSL: Trajectory-Enhanced Semi-Supervised 3D Object Detection [59.498894868956306]
  Pseudo-labeling approaches to semi-supervised learning adopt a teacher-student framework. We leverage pre-trained motion-forecasting models to generate object trajectories on pseudo-labeled data. Our approach improves pseudo-label quality in two distinct manners.
  arXiv  Detail & Related papers  (2024-09-17T05:35:00Z)
• Towards Modality-agnostic Label-efficient Segmentation with Entropy-Regularized Distribution Alignment [62.73503467108322]
  This topic is widely studied in 3D point cloud segmentation due to the difficulty of annotating point clouds densely. Until recently, pseudo-labels have been widely employed to facilitate training with limited ground-truth labels. Existing pseudo-labeling approaches could suffer heavily from the noises and variations in unlabelled data. We propose a novel learning strategy to regularize the pseudo-labels generated for training, thus effectively narrowing the gaps between pseudo-labels and model predictions.
  arXiv  Detail & Related papers  (2024-08-29T13:31:15Z)
• Label-Efficient 3D Brain Segmentation via Complementary 2D Diffusion Models with Orthogonal Views [10.944692719150071]
  We propose a novel 3D brain segmentation approach using complementary 2D diffusion models. Our goal is to achieve reliable segmentation quality without requiring complete labels for each individual subject.
  arXiv  Detail & Related papers  (2024-07-17T06:14:53Z)
• WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments [33.25040383298019]
  $WildScenes$ is a bi-modal benchmark dataset consisting of high-resolution 2D images and dense 3D LiDAR point clouds. The data is trajectory-centric with accurate localization and globally aligned point clouds. Our 3D semantic labels are obtained via an efficient, automated process that transfers the human-annotated 2D labels from multiple views into 3D point cloud sequences.
  arXiv  Detail & Related papers  (2023-12-23T22:27:40Z)
• Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation [46.01433705072047]
  We find two types of latent labels behind the displayed label embedded in LiDAR and image data. We propose a novel augmentation, Cylinder-Mix, which is able to augment more yet reliable samples for training. We also propose the Instance Position-scale Learning (IPSL) Module to learn and fuse the information of instance position and scale.
  arXiv  Detail & Related papers  (2023-12-13T15:56:24Z)
• Self-Supervised 3D Scene Flow Estimation and Motion Prediction using Local Rigidity Prior [100.98123802027847]
  We investigate self-supervised 3D scene flow estimation and class-agnostic motion prediction on point clouds. We generate pseudo scene flow labels for self-supervised learning through piecewise rigid motion estimation. Our method achieves new state-of-the-art performance in self-supervised scene flow learning.
  arXiv  Detail & Related papers  (2023-10-17T14:06:55Z)
• Unsupervised 3D registration through optimization-guided cyclical self-training [71.75057371518093]
  State-of-the-art deep learning-based registration methods employ three different learning strategies. We propose a novel self-supervised learning paradigm for unsupervised registration, relying on self-training. We evaluate the method for abdomen and lung registration, consistently surpassing metric-based supervision and outperforming diverse state-of-the-art competitors.
  arXiv  Detail & Related papers  (2023-06-29T14:54:10Z)
• TempNet: Temporal Attention Towards the Detection of Animal Behaviour in Videos [63.85815474157357]
  We propose an efficient computer vision- and deep learning-based method for the detection of biological behaviours in videos. TempNet uses an encoder bridge and residual blocks to maintain model performance with a two-staged, spatial, then temporal, encoder. We demonstrate its application to the detection of sablefish (Anoplopoma fimbria) startle events.
  arXiv  Detail & Related papers  (2022-11-17T23:55:12Z)
• LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds [62.49198183539889]
  We propose a label-efficient semantic segmentation pipeline for outdoor scenes with LiDAR point clouds. Our method co-designs an efficient labeling process with semi/weakly supervised learning. Our proposed method is even highly competitive compared to the fully supervised counterpart with 100% labels.
  arXiv  Detail & Related papers  (2022-10-14T19:13:36Z)
• Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning [59.64695628433855]
  We propose a novel cross-modality weakly supervised method for 3D segmentation, incorporating complementary information from unlabeled images. Basically, we design a dual-branch network equipped with an active labeling strategy, to maximize the power of tiny parts of labels. Our method even outperforms the state-of-the-art fully supervised competitors with less than 1% actively selected annotations.
  arXiv  Detail & Related papers  (2022-09-16T07:59:04Z)
• Collaborative Propagation on Multiple Instance Graphs for 3D Instance Segmentation with Single-point Supervision [63.429704654271475]
  We propose a novel weakly supervised method RWSeg that only requires labeling one object with one point. With these sparse weak labels, we introduce a unified framework with two branches to propagate semantic and instance information. Specifically, we propose a Cross-graph Competing Random Walks (CRW) algorithm that encourages competition among different instance graphs.
  arXiv  Detail & Related papers  (2022-08-10T02:14:39Z)
• Applications of Deep Learning in Fish Habitat Monitoring: A Tutorial and Survey [1.9249287163937976]
  Deep learning (DL) is a cutting-edge AI technology that has demonstrated unprecedented performance in analysing visual data. In this paper, we provide a tutorial that covers the key concepts of DL, which help the reader grasp a high-level understanding of how DL works. The tutorial also explains a step-by-step procedure on how DL algorithms should be developed for challenging applications such as underwater fish monitoring.
  arXiv  Detail & Related papers  (2022-06-11T01:59:54Z)
• Overcoming Annotation Bottlenecks in Underwater Fish Segmentation: A Robust Self-Supervised Learning Approach [3.0516727053033392]
  This paper introduces a novel self-supervised learning approach for fish segmentation using Deep Learning. Our model, trained without manual annotation, learns robust and generalizable representations by aligning features across augmented views. We demonstrate its effectiveness on three challenging underwater video datasets: DeepFish, Seagrass, and YouTube-VOS.
  arXiv  Detail & Related papers  (2022-06-11T01:20:48Z)
• Unlocking the potential of deep learning for marine ecology: overview, applications, and outlook [8.3226670069051]
  This paper aims to bridge the gap between marine ecologists and computer scientists. We provide insight into popular deep learning approaches for ecological data analysis in plain language. We illustrate challenges and opportunities through established and emerging applications of deep learning to marine ecology.
  arXiv  Detail & Related papers  (2021-09-29T21:59:16Z)
• Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation [90.87105131054419]
  We present a framework for semi-supervised semantic segmentation, which is enhanced by self-supervised monocular depth estimation from unlabeled image sequences. We validate the proposed model on the Cityscapes dataset, where all three modules demonstrate significant performance gains.
  arXiv  Detail & Related papers  (2020-12-19T21:18:03Z)
• Movement Tracks for the Automatic Detection of Fish Behavior in Videos [63.85815474157357]
  We offer a dataset of sablefish (Anoplopoma fimbria) startle behaviors in underwater videos, and investigate the use of deep learning (DL) methods for behavior detection on it. Our proposed detection system identifies fish instances using DL-based frameworks, determines trajectory tracks, derives novel behavior-specific features, and employs Long Short-Term Memory (LSTM) networks to identify startle behavior in sablefish.
  arXiv  Detail & Related papers  (2020-11-28T05:51:19Z)
• A Realistic Fish-Habitat Dataset to Evaluate Algorithms for Underwater Visual Analysis [2.6476746128312194]
  We present DeepFish as a benchmark suite with a large-scale dataset to train and test methods for several computer vision tasks. The dataset consists of approximately 40 thousand images collected underwater from 20 greenhabitats in the marine-environments of tropical Australia. Our experiments provide an in-depth analysis of the dataset characteristics, and the performance evaluation of several state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-08-28T12:20:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.