Agent-Centric Animal Pose Forecasting

• URL: http://arxiv.org/abs/2607.19548v1
• Date: Tue, 21 Jul 2026 19:48:21 GMT
• Title: Agent-Centric Animal Pose Forecasting
• Abstract summary: We introduce a framework for training agent-centric autoregressive models of animal behavior.<n>We show that trained models capture the distribution of social behavior in groups of courting Drosophila.<n>We release a general-purpose library focused on the composable sequences of operations that translate between these representations.
• Score: 0.1494633072572191
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding animal behavior at an algorithmic level -- what animals attend to, how they form internal models and plans, and how this maps to action -- remains a central challenge in neuroscience and ethology. Data-driven generative models offer a path toward this understanding. We introduce a framework for training agent-centric autoregressive models of animal behavior from tracked pose, applicable to single animals and to groups in which each agent senses and responds to its conspecifics. Our models input egocentric sensory observations and output egocentric movements, mirroring the biological constraint that animals observe and act on the world from their own reference frame. Social behavior emerges from agents independently sensing and responding to one another. This agent-centric formulation requires managing many parallel representations of the same data, along with ML-specific transformations like discretization. We release a general-purpose library focused on the composable sequences of operations that translate between these representations. We show that trained models capture the distribution of social behavior in groups of courting Drosophila, and our library includes quantitative tools for measuring fit. We demonstrate how the library supports systematic comparison across input and output representations and that it adapts straightforwardly to a new domain.

Related papers

• What Does Animal Re-Identification Learn? Linear Biological Concepts and Their Origins in Visual Representations [52.11506532642883]
  We show that sex and age emerge as linear directions that generalize to held-out individuals.<n>We also show that sex direction is causally used by the model, flipping a significant fraction of predictions to the opposite sex.
  arXiv  Detail & Related papers  (2026-09-05T10:46:27Z)
• kabr-tools: Automated Framework for Multi-Species Behavioral Monitoring [4.303185550812535]
  We present kabr-tools, an open-source package for automated multi-species behavioral monitoring.<n>This framework integrates drone-based video with machine learning systems to extract behavioral, social, and spatial metrics from wildlife footage.<n>Compared to ground-based methods, drone-based observations significantly improved behavioral granularity, reducing visibility loss by 15%.
  arXiv  Detail & Related papers  (2025-10-02T14:03:55Z)
• EgoPet: Egomotion and Interaction Data from an Animal's Perspective [82.7192364237065]
  We introduce a dataset of pet egomotion imagery with diverse examples of simultaneous egomotion and multi-agent interaction. EgoPet offers a radically distinct perspective from existing egocentric datasets of humans or vehicles. We define two in-domain benchmark tasks that capture animal behavior, and a third benchmark to assess the utility of EgoPet as a pretraining resource to robotic quadruped locomotion.
  arXiv  Detail & Related papers  (2024-04-15T17:59:47Z)
• Behaviour Modelling of Social Animals via Causal Structure Discovery and Graph Neural Networks [15.542220566525021]
  We propose a method to build behavioural models using causal structure discovery and graph neural networks for time series. We apply this method to a mob of meerkats in a zoo environment and study its ability to predict future actions.
  arXiv  Detail & Related papers  (2023-12-21T23:34:08Z)
• LISBET: a machine learning model for the automatic segmentation of social behavior motifs [0.0]
  We introduce LISBET (LISBET Is a Social BEhavior Transformer), a machine learning model for detecting and segmenting social interactions. Using self-supervised learning on body tracking data, our model eliminates the need for extensive human annotation. In vivo electrophysiology revealed distinct neural signatures in the Ventral Tegmental Area corresponding to motifs identified by our model.
  arXiv  Detail & Related papers  (2023-11-07T15:35:17Z)
• Homomorphism Autoencoder -- Learning Group Structured Representations from Observed Transitions [51.71245032890532]
  We propose methods enabling an agent acting upon the world to learn internal representations of sensory information consistent with actions that modify it. In contrast to existing work, our approach does not require prior knowledge of the group and does not restrict the set of actions the agent can perform.
  arXiv  Detail & Related papers  (2022-07-25T11:22:48Z)
• Drawing out of Distribution with Neuro-Symbolic Generative Models [49.79371715591122]
  Drawing out of Distribution is a neuro-symbolic generative model of stroke-based drawing. DooD operates directly on images, requires no supervision or expensive test-time inference. We evaluate DooD on its ability to generalise across both data and tasks.
  arXiv  Detail & Related papers  (2022-06-03T21:40:22Z)
• SuperAnimal pretrained pose estimation models for behavioral analysis [42.206265576708255]
  Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop unified foundation models.
  arXiv  Detail & Related papers  (2022-03-14T18:46:57Z)
• Beyond Tracking: Using Deep Learning to Discover Novel Interactions in Biological Swarms [3.441021278275805]
  We propose training deep network models to predict system-level states directly from generic graphical features from the entire view. Because the resulting predictive models are not based on human-understood predictors, we use explanatory modules. This represents an example of augmented intelligence in behavioral ecology -- knowledge co-creation in a human-AI team.
  arXiv  Detail & Related papers  (2021-08-20T22:50:41Z)
• Learning interaction rules from multi-animal trajectories via augmented behavioral models [8.747278400158718]
  Granger causality is a practical framework for analyzing the interactions from observed time-series data. This framework ignores the structures of the generative process in animal behaviors. We propose a new framework for learning Granger causality from multi-animal trajectories.
  arXiv  Detail & Related papers  (2021-07-12T11:33:56Z)
• Muti-view Mouse Social Behaviour Recognition with Deep Graphical Model [124.26611454540813]
  Social behaviour analysis of mice is an invaluable tool to assess therapeutic efficacy of neurodegenerative diseases. Because of the potential to create rich descriptions of mouse social behaviors, the use of multi-view video recordings for rodent observations is increasingly receiving much attention. We propose a novel multiview latent-attention and dynamic discriminative model that jointly learns view-specific and view-shared sub-structures.
  arXiv  Detail & Related papers  (2020-11-04T18:09:58Z)
• Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals [56.65062746564091]
  We aim to capture the pose of neuroscience model organisms, without using any manual supervision, to study how neural circuits orchestrate behaviour. Our key contribution is the explicit and independent modeling of appearance, shape and poses in an unpaired image translation framework. We demonstrate improved pose estimation accuracy on Drosophila melanogaster (fruit fly), Caenorhabditis elegans (worm) and Danio rerio (zebrafish)
  arXiv  Detail & Related papers  (2020-01-23T15:34:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.