BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning

• URL: http://arxiv.org/abs/2603.12176v1
• Date: Thu, 12 Mar 2026 17:09:20 GMT
• Title: BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning
• Abstract summary: We present BehaviorVLM, a unified framework for pose estimation and behavioral understanding.<n>For pose estimation, we propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning.<n>For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery.
• Score: 12.25724335471528
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM, a unified vision-language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained Vision-Language Models (VLMs) through detailed, explicit, and verifiable reasoning steps. For pose estimation, we leverage quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip video captioning, and LLM-based reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior.

Related papers

• Detecting and Controlling Sycophancy with Cascading Linear Features [7.13923185919936]
  We present an iterative data generation pipeline that isolates cascading linear features responsible for a behavior.<n>We focus on detecting and steering away from sycophancy -- the tendency of language models to prioritize user validation.
  arXiv  Detail & Related papers  (2026-06-23T20:10:53Z)
• QuoVLA: Quotient Space for Vision-Language-Action Models [51.02329790939691]
  Vision-Language-Action (VLA) models commonly adapt pretrained VLM latents to robot control by mapping visual observations and language instructions to continuous actions.<n>Our textitQuotient Theory for VLA shows that pretrained VLM latents are not action-insufficient but action-sufficient.<n>We propose QuoVLA, a quotient-space framework for VLA that compresses pretrained VLM latents into action-sufficient representations.
  arXiv  Detail & Related papers  (2026-05-24T06:28:53Z)
• GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization [101.37117235471709]
  We introduce GuidedVLA, a framework designed to guide the action generation to focus on task-relevant factors.<n>Our core insight is to treat the action decoder not as a monolithic learner, but as an assembly of functional components.<n>Our results suggest that explicitly guiding action-decoder learning is a promising direction for building more robust and general VLA models.
  arXiv  Detail & Related papers  (2026-05-12T16:38:40Z)
• Behavioral Intelligence Platforms: From Event Streams to Autonomous Insight via Probabilistic Journey Graphs, Behavioral Knowledge Extraction, and Grounded Language Generation [0.0]
  We present the Behavioral Intelligence Platform (BIP), a system architecture that transforms raw event streams into automatically generated insights.<n>BIP consists of four layers. First, Normalization and State Derivation (NSD) standardizes events and maps them to a semantic state hierarchy.<n>Second, a Behavioral Graph Engine (BGE) models user journeys as absorbing Markov chains and computes transition probabilities, removal effects, and path quality metrics.<n>Third, a Behavioral Knowledge Graph (BKG) and Detector System convert graph outputs into grounded behavioral facts and identify behavioral phenomena.
  arXiv  Detail & Related papers  (2026-03-12T16:47:43Z)
• From Latent Signals to Reflection Behavior: Tracing Meta-Cognitive Activation Trajectory in R1-Style LLMs [48.33546389897804]
  R1-style LLMs have attracted growing attention for their capacity for self-reflection, yet the internal mechanisms underlying such behavior remain unclear.<n>Using the logit lens to read out token-level semantics, we uncover a structured progression.<n>Our findings suggest a human-like metacognitive process-progressing from latent monitoring, to discourse-level regulation, and to finally overt self-reflection.
  arXiv  Detail & Related papers  (2026-02-02T11:58:24Z)
• \textsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• CBMAS: Cognitive Behavioral Modeling via Activation Steering [5.131778762865578]
  Large language models (LLMs) often encode cognitive behaviors unpredictably across prompts, layers, and contexts.<n>We present CBMAS, a diagnostic framework for continuous activation steering.
  arXiv  Detail & Related papers  (2026-01-03T13:04:14Z)
• Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process [66.38541693477181]
  We propose an unsupervised framework for discovering reasoning vectors, which we define as directions in the activation space that encode distinct reasoning behaviors.<n>By segmenting chain-of-thought traces into sentence-level'steps', we uncover disentangled features corresponding to interpretable behaviors such as reflection and backtracking.<n>We demonstrate the ability to control response confidence by identifying confidence-related vectors in the SAE decoder space.
  arXiv  Detail & Related papers  (2025-12-30T05:09:11Z)
• Controllable Human-Object Interaction Synthesis [77.56877961681462]
  We propose Controllable Human-Object Interaction Synthesis (CHOIS) to generate synchronized object motion and human motion in 3D scenes. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.
  arXiv  Detail & Related papers  (2023-12-06T21:14:20Z)
• Learning What and Where -- Unsupervised Disentangling Location and Identity Tracking [0.44040106718326594]
  We introduce an unsupervisedd LOCation and Identity tracking system (Loci) Inspired by the dorsal-ventral pathways in the brain, Loci tackles the what-and-where binding problem by means of a self-supervised segregation mechanism. Loci may set the stage for deeper, explanation-oriented video processing.
  arXiv  Detail & Related papers  (2022-05-26T13:30:14Z)
• Inference of Affordances and Active Motor Control in Simulated Agents [0.5161531917413706]
  We introduce an output-probabilistic, temporally predictive, modular artificial neural network architecture. We show that our architecture develops latent states that can be interpreted as affordance maps. In combination with active inference, we show that flexible, goal-directed behavior can be invoked.
  arXiv  Detail & Related papers  (2022-02-23T14:13:04Z)
• Learning Asynchronous and Sparse Human-Object Interaction in Videos [56.73059840294019]
  Asynchronous-Sparse Interaction Graph Networks (ASSIGN) is able to automatically detect the structure of interaction events associated with entities in a video scene. ASSIGN is tested on human-object interaction recognition and shows superior performance in segmenting and labeling of human sub-activities and object affordances from raw videos.
  arXiv  Detail & Related papers  (2021-03-03T23:43:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.