A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

• URL: http://arxiv.org/abs/2407.16727v1
• Date: Tue, 23 Jul 2024 14:22:16 GMT
• Title: A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms
• Authors: Ari Blau, Evan S Schaffer, Neeli Mishra, Nathaniel J Miska, The International Brain Laboratory, Liam Paninski, Matthew R Whiteway,
• Abstract summary: We introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information perform the best on our supervised metrics across all datasets.
• Score: 3.597220870252727
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms -- which include tree-based models, deep neural networks, and graphical models -- differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species -- fly, mouse, and human -- we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.

Related papers

• Interpretable statistical representations of neural population dynamics and geometry [4.459704414303749]
  We introduce MARBLE, a fully unsupervised geometric deep learning framework for the data-driven representation of non-linear dynamics. We show that MARBLE can infer latent representations that are highly interpretable in terms of global system variables. We also show that MARBLE representations are consistent across neural networks and animals so that they can be used to compare cognitive computations or train universal decoders.
  arXiv  Detail & Related papers  (2023-04-06T21:11:04Z)
• Overcoming the Domain Gap in Neural Action Representations [60.47807856873544]
  3D pose data can now be reliably extracted from multi-view video sequences without manual intervention. We propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations. To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions.
  arXiv  Detail & Related papers  (2021-12-02T12:45:46Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• Learning Neural Causal Models with Active Interventions [83.44636110899742]
  We introduce an active intervention-targeting mechanism which enables a quick identification of the underlying causal structure of the data-generating process. Our method significantly reduces the required number of interactions compared with random intervention targeting. We demonstrate superior performance on multiple benchmarks from simulated to real-world data.
  arXiv  Detail & Related papers  (2021-09-06T13:10:37Z)
• Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms [3.441021278275805]
  We propose training deep network models to predict system-level states directly from generic graphical features from the entire view. Because the resulting predictive models are not based on human-understood predictors, we use explanatory modules. This represents an example of augmented intelligence in behavioral ecology -- knowledge co-creation in a human-AI team.
  arXiv  Detail & Related papers  (2021-08-20T22:50:41Z)
• Learning interaction rules from multi-animal trajectories via augmented behavioral models [8.747278400158718]
  Granger causality is a practical framework for analyzing the interactions from observed time-series data. This framework ignores the structures of the generative process in animal behaviors. We propose a new framework for learning Granger causality from multi-animal trajectories.
  arXiv  Detail & Related papers  (2021-07-12T11:33:56Z)
• Unsupervised Discovery of the Long-Tail in Instance Segmentation Using Hierarchical Self-Supervision [3.841232411073827]
  We propose a method that can perform unsupervised discovery of long-tail categories in instance segmentation. Our model is able to discover novel and more fine-grained objects than the common categories. We show that the model achieves competitive quantitative results on LVIS as compared to the supervised and partially supervised methods.
  arXiv  Detail & Related papers  (2021-04-02T22:05:03Z)
• Unsupervised Speech Representation Learning for Behavior Modeling using Triplet Enhanced Contextualized Networks [28.957236790411585]
  We exploit the stationary properties of human behavior within an interaction and present a representation learning method to capture behavioral information from speech. We present an encoder-decoder based Deep Contextualized Network (DCN) as well as a Triplet-Enhanced DCN (TE-DCN) framework to capture the behavioral context.
  arXiv  Detail & Related papers  (2021-04-01T22:44:23Z)
• Visual Distant Supervision for Scene Graph Generation [66.10579690929623]
  Scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. We propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines.
  arXiv  Detail & Related papers  (2021-03-29T06:35:24Z)
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning [50.24174211654775]
  We present a novel contrastive self-supervised learning framework for anomaly detection on attributed networks. Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair. A graph neural network-based contrastive learning model is proposed to learn informative embedding from high-dimensional attributes and local structure.
  arXiv  Detail & Related papers  (2021-02-27T03:17:20Z)
• Learning to Abstract and Predict Human Actions [60.85905430007731]
  We model the hierarchical structure of human activities in videos and demonstrate the power of such structure in action prediction. We propose Hierarchical-Refresher-Anticipator, a multi-level neural machine that can learn the structure of human activities by observing a partial hierarchy of events and roll-out such structure into a future prediction in multiple levels of abstraction.
  arXiv  Detail & Related papers  (2020-08-20T23:57:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.