TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life

• URL: http://arxiv.org/abs/2603.26128v1
• Date: Fri, 27 Mar 2026 07:22:43 GMT
• Title: TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life
• Abstract summary: TaxaAdapter is a simple and lightweight approach that incorporates Vision Taxonomy Models (VTMs) to guide fine-grained species generation.<n>Our method injects VTM embeddings into a frozen text-to-image diffusion model, improving species-level fidelity.<n>Our results highlight that VTMs are a key ingredient for scalable, fine-grained species generation.
• Score: 39.89855645414284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurately generating images across the Tree of Life is difficult: there are over 10M distinct species on Earth, many of which differ only by subtle visual traits. Despite the remarkable progress in text-to-image synthesis, existing models often fail to capture the fine-grained visual cues that define species identity, even when their outputs appear photo-realistic. To this end, we propose TaxaAdapter, a simple and lightweight approach that incorporates Vision Taxonomy Models (VTMs) such as BioCLIP to guide fine-grained species generation. Our method injects VTM embeddings into a frozen text-to-image diffusion model, improving species-level fidelity while preserving flexible text control over attributes such as pose, style, and background. Extensive experiments demonstrate that TaxaAdapter consistently improves morphology fidelity and species-identity accuracy over strong baselines, with a cleaner architecture and training recipe. To better evaluate these improvements, we also introduce a multimodal Large Language Model-based metric that summarizes trait-level descriptions from generated and real images, providing a more interpretable measure of morphological consistency. Beyond this, we observe that TaxaAdapter exhibits strong generalization capabilities, enabling species synthesis in challenging regimes such as few-shot species with only a handful of training images and even species unseen during training. Overall, our results highlight that VTMs are a key ingredient for scalable, fine-grained species generation.

Related papers

• TreeAdapter: Hierarchical Taxonomy-Guided Adapter Composition for Fine-Grained Species Image Generation [4.373125385645128]
  TreeAdapter is a novel framework that explicitly leverages hierarchical taxonomic data.<n>We introduce a two-stage training paradigm where ancestor adapters are optimized to model only the residual visual features unexplained by their descendants.<n>TreeAdapter achieves state-of-the-art fine-grained generation quality, outperforming both general-purpose and domain-specific baselines.
  arXiv  Detail & Related papers  (2026-07-27T09:47:55Z)
• Leveraging Image Generators to Address Training Data Scarcity: The Gen4Regen Dataset for Forest Regeneration Mapping [6.12592712646272]
  This paper addresses the dual challenges of data scarcity and extreme class imbalance in the semantic segmentation of fine-grained forest regeneration species.<n>We leverage the large-scale vision-language Nano Banana Pro model to simultaneously generate high-fidelity images and their corresponding pixel-aligned semantic masks from prompts.<n>Our methodology integrates real-world data with AI-generated images, highlighting that AI-generated data is highly complementary to real-world data.
  arXiv  Detail & Related papers  (2026-05-07T03:28:56Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• TaxaDiffusion: Progressively Trained Diffusion Model for Fine-Grained Species Generation [27.543784765817513]
  TaxaDiffusion is a taxonomy-informed training framework for diffusion models.<n>It generates fine-grained animal images with high morphological and identity accuracy.
  arXiv  Detail & Related papers  (2025-06-02T17:43:55Z)
• Taxonomic Reasoning for Rare Arthropods: Combining Dense Image Captioning and RAG for Interpretable Classification [12.923336716880506]
  We integrate image captioning and retrieval-augmented generation (RAG) with large language models (LLMs) to enhance biodiversity monitoring.<n>Our findings highlight the potential for modern vision-language AI pipelines to support biodiversity conservation initiatives.
  arXiv  Detail & Related papers  (2025-03-13T21:18:10Z)
• DAMamba: Vision State Space Model with Dynamic Adaptive Scan [51.81060691414399]
  State space models (SSMs) have recently garnered significant attention in computer vision.<n>We propose Dynamic Adaptive Scan (DAS), a data-driven method that adaptively allocates scanning orders and regions.<n>Based on DAS, we propose the vision backbone DAMamba, which significantly outperforms current state-of-the-art vision Mamba models in vision tasks.
  arXiv  Detail & Related papers  (2025-02-18T08:12:47Z)
• MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classification [36.59100450109841]
  Whole slide pathology image classification presents challenges due to gigapixel image sizes and limited annotation labels.<n>This paper introduces a prompt learning method to adapt large vision-language models for few-shot pathology classification.
  arXiv  Detail & Related papers  (2025-02-11T09:42:13Z)
• Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection [106.39544368711427]
  We study the problem of generalizable synthetic image detection, aiming to detect forgery images from diverse generative methods. We present a novel forgery-aware adaptive transformer approach, namely FatFormer. Our approach tuned on 4-class ProGAN data attains an average of 98% accuracy to unseen GANs, and surprisingly generalizes to unseen diffusion models with 95% accuracy.
  arXiv  Detail & Related papers  (2023-12-27T17:36:32Z)
• SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification [84.05253637260743]
  We propose a new framework, named Semantic-guided Visual Adapting (SgVA), to extend vision-language pre-trained models. SgVA produces discriminative task-specific visual features by comprehensively using a vision-specific contrastive loss, a cross-modal contrastive loss, and an implicit knowledge distillation. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.
  arXiv  Detail & Related papers  (2022-11-28T14:58:15Z)
• Deep Low-Shot Learning for Biological Image Classification and Visualization from Limited Training Samples [52.549928980694695]
  In situ hybridization (ISH) gene expression pattern images from the same developmental stage are compared. labeling training data with precise stages is very time-consuming even for biologists. We propose a deep two-step low-shot learning framework to accurately classify ISH images using limited training images.
  arXiv  Detail & Related papers  (2020-10-20T06:06:06Z)
• Two-View Fine-grained Classification of Plant Species [66.75915278733197]
  We propose a novel method based on a two-view leaf image representation and a hierarchical classification strategy for fine-grained recognition of plant species. A deep metric based on Siamese convolutional neural networks is used to reduce the dependence on a large number of training samples and make the method scalable to new plant species.
  arXiv  Detail & Related papers  (2020-05-18T21:57:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.