BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

• URL: http://arxiv.org/abs/2603.23883v1
• Date: Wed, 25 Mar 2026 03:15:04 GMT
• Title: BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment
• Abstract summary: Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology.<n>We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications.
• Score: 42.72898704470983
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/

Related papers

• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• BioMiner: A Multi-modal System for Automated Mining of Protein-Ligand Bioactivity Data from Literature [53.894504720119805]
  We introduce BioMiner, a multi-modal extraction framework for bioactivity data extraction.<n>Within BioMiner, bioactivity semantics are inferred through direct reasoning, while chemical structures are resolved via a chemical-structure-grounded visual semantic reasoning paradigm.<n>For rigorous evaluation and method development, we establish a benchmark comprising 16,457 bioactivity entries curated from 500 publications.
  arXiv  Detail & Related papers  (2026-04-23T10:11:56Z)
• The iNaturalist Sounds Dataset [60.157076990024606]
  iNatSounds is a collection of 230,000 audio files capturing sounds from over 5,500 species, contributed by more than 27,000 recordists worldwide.<n>The dataset encompasses sounds from birds, mammals, insects, reptiles, and amphibians, with audio and species labels derived from observations submitted to iNaturalist.<n>We envision models trained on this data powering next-generation public engagement applications, and assisting biologists, ecologists, and land use managers in processing large audio collections.
  arXiv  Detail & Related papers  (2025-05-31T02:07:37Z)
• BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learning [60.80381372245902]
  We find emergent behaviors in biological vision models via large-scale contrastive vision-language training.<n>We train BioCLIP 2 on TreeOfLife-200M to distinguish different species.<n>We identify emergent properties in the learned embedding space of BioCLIP 2.
  arXiv  Detail & Related papers  (2025-05-29T17:48:20Z)
• CrypticBio: A Large Multimodal Dataset for Visually Confusing Biodiversity [3.73232466691291]
  We present CrypticBio, the largest publicly available dataset of visually confusing species.<n>Criticized from real-world trends in species misidentification among community annotators of iNaturalist, CrypticBio contains 52K unique cryptic groups spanning 67K species.
  arXiv  Detail & Related papers  (2025-05-16T14:35:56Z)
• BioD2C: A Dual-level Semantic Consistency Constraint Framework for Biomedical VQA [5.840467499436581]
  We propose BioD2C: a novel Dual-level Semantic Consistency Constraint Framework for Biomedical VQA.<n>BioD2C achieves dual-level semantic interaction alignment at both the model and feature levels, enabling the model to adaptively learn visual features based on the question.<n>In this work, we establish a new dataset, BioVGQ, to address inherent biases in prior datasets by filtering manually-altered images and aligning question-answer pairs with multimodal context.
  arXiv  Detail & Related papers  (2025-03-04T10:39:42Z)
• NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics [35.72581102737726]
  We present NatureLM-audio, the first audio-language foundation model specifically designed for bioacoustics.<n>Our training dataset consists of carefully curated text-audio pairs spanning bioacoustics, speech, and music.<n>We demonstrate successful transfer of learned representations from music and speech to bioacoustics, and our model shows promising generalization to unseen taxa and tasks.
  arXiv  Detail & Related papers  (2024-11-11T18:01:45Z)
• BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity [14.949271003068107]
  BioTrove is the largest dataset designed to advance AI applications in biodiversity.<n>It contains 161.9 million images, offering unprecedented scale and diversity from three primary kingdoms.<n>Each image is annotated with scientific names, taxonomic hierarchies, and common names.
  arXiv  Detail & Related papers  (2024-06-25T17:09:54Z)
• animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics [2.1019401515721583]
  animal2vec is an interpretable large transformer model that learns from unlabeled audio and refines its understanding with labeled data. Meerkat Audio Transcripts is the largest labeled dataset on non-human terrestrial mammals. Our model outperforms existing methods on MeerKAT and the publicly available NIPS4Bplus birdsong dataset.
  arXiv  Detail & Related papers  (2024-06-03T12:11:01Z)
• Transferable Models for Bioacoustics with Human Language Supervision [0.0]
  BioLingual is a new model for bioacoustics based on contrastive language-audio pretraining. It can identify over a thousand species' calls across taxa, complete bioacoustic tasks zero-shot, and retrieve animal vocalization recordings from natural text queries.
  arXiv  Detail & Related papers  (2023-08-09T14:22:18Z)
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs [46.87322157229728]
  We present PMC-15M, a novel dataset that is two orders of magnitude larger than existing biomedical multimodal datasets.<n> PMC-15M contains 15 million biomedical image-text pairs collected from 4.4 million scientific articles.<n>Based on PMC-15M, we have pretrained BiomedCLIP, a multimodal foundation model, with domain-specific adaptations tailored to biomedical vision-language processing.
  arXiv  Detail & Related papers  (2023-03-02T02:20:04Z)
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing [53.89917396428747]
  Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. We explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model.
  arXiv  Detail & Related papers  (2023-01-11T16:35:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.