Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance

• URL: http://arxiv.org/abs/2608.18602v1
• Date: Wed, 19 Aug 2026 06:44:42 GMT
• Title: Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance
• Abstract summary: We propose a new approach to working with imperfect tracking predictions through an interactive prediction correction workflow.<n>We tailor a sonar fish tracking dataset as an initial proof of concept.<n>We find that Molmo2Fish achieves high performance on fish tracking and track correction tasks.
• Score: 6.626960192028637
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer vision is increasingly used to automate recognition tasks in large ecological datasets, but more complex tasks such as multi-object tracking continue to pose challenges. As researchers seek to incorporate vision models in ecology workflows, various lines of research have explored how to make imperfect predictions useful through human-in-the-loop processes. We propose a new approach to working with imperfect tracking predictions through an interactive prediction correction workflow taking place as a conversation with a multimodal large language model, which we tailor to a sonar fish tracking dataset as an initial proof of concept. We investigate the performance of the tool, Molmo2Fish, across guided and unguided tasks, correcting its own predicted tracks and external tracks. We find that Molmo2Fish achieves high performance on fish tracking and track correction tasks, but there is still much room to improve on incorporating natural language guidance. The code and data are publicly available at https://github.com/tidalove/molmo2fish.

Related papers

• Evolution of Concepts in Language Model Pre-Training [53.994470178155105]
  We track linear interpretable feature evolution across pre-training snapshots using a sparse dictionary learning method called crosscoders.<n>We find that most features begin to form around a specific point, while more complex patterns emerge in later training stages.
  arXiv  Detail & Related papers  (2025-09-21T18:53:12Z)
• Benchmarking pig detection and tracking under diverse and challenging conditions [1.865175170209582]
  We curated two datasets: PigDetect for object detection and PigTrack for multi-object tracking.<n>For object detection, we show that challenging training images improve detection beyond what is achievable with randomly sampled images alone.<n>For multi-object tracking, we observed that SORT-based methods achieve superior detection performance compared to end-to-end trainable models.
  arXiv  Detail & Related papers  (2025-07-22T14:36:51Z)
• A Framework for Multi-View Multiple Object Tracking using Single-View Multi-Object Trackers on Fish Data [0.559239450391449]
  This thesis adapts state-of-the-art single-view MOT models, FairMOT and YOLOv8, for underwater fish detecting and tracking in ecological studies.<n>The proposed framework detects fish entities with a relative accuracy of 47% and employs stereo-matching techniques to produce a novel 3D output.
  arXiv  Detail & Related papers  (2025-05-22T18:12:08Z)
• AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via GRPO [0.0]
  Large Language Models (LLMs) have demonstrated impressive capabilities in language processing, yet they often struggle with tasks requiring visual spatial reasoning.<n>We introduce a novel two-stage training framework designed to equip standard LLMs with visual reasoning abilities for maze navigation.
  arXiv  Detail & Related papers  (2025-02-20T16:05:18Z)
• DiSciPLE: Learning Interpretable Programs for Scientific Visual Discovery [61.02102713094486]
  Good interpretation is important in scientific reasoning, as it allows for better decision-making.<n>This paper introduces an automatic way of obtaining such interpretable-by-design models, by learning programs that interleave neural networks.<n>We propose DiSciPLE an evolutionary algorithm that leverages common sense and prior knowledge of large language models (LLMs) to create Python programs explaining visual data.
  arXiv  Detail & Related papers  (2025-02-14T10:26:14Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• Octopus: Embodied Vision-Language Programmer from Environmental Feedback [58.04529328728999]
  Embodied vision-language models (VLMs) have achieved substantial progress in multimodal perception and reasoning. To bridge this gap, we introduce Octopus, an embodied vision-language programmer that uses executable code generation as a medium to connect planning and manipulation. Octopus is designed to 1) proficiently comprehend an agent's visual and textual task objectives, 2) formulate intricate action sequences, and 3) generate executable code.
  arXiv  Detail & Related papers  (2023-10-12T17:59:58Z)
• Policy Pre-training for End-to-end Autonomous Driving via Self-supervised Geometric Modeling [96.31941517446859]
  We propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos. In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.
  arXiv  Detail & Related papers  (2023-01-03T08:52:49Z)
• Palm up: Playing in the Latent Manifold for Unsupervised Pretraining [31.92145741769497]
  We propose an algorithm that exhibits an exploratory behavior whilst it utilizes large diverse datasets. Our key idea is to leverage deep generative models that are pretrained on static datasets and introduce a dynamic model in the latent space. We then employ an unsupervised reinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data.
  arXiv  Detail & Related papers  (2022-10-19T22:26:12Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• Leveraging Pre-Trained Language Models to Streamline Natural Language Interaction for Self-Tracking [25.28975864365579]
  We propose a novel NLP task for self-tracking that extracts close- and open-ended information from a retrospective activity log. The framework augments the prompt using synthetic samples to transform the task into 10-shot learning, to address a cold-start problem in bootstrapping a new tracking topic.
  arXiv  Detail & Related papers  (2022-05-31T01:58:04Z)
• A Survey of Fish Tracking Techniques Based on Computer Vision [11.994865945394139]
  This paper presents a review of the advancements of fish tracking technologies over the past seven years-2023. It explores diverse fish tracking techniques with an emphasis on fundamental localization and tracking methods. It also summarizes open-source datasets, evaluation metrics, challenges, and applications in fish tracking research.
  arXiv  Detail & Related papers  (2021-10-06T07:46:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.