VR-RAG: Open-vocabulary Species Recognition with RAG-Assisted Large Multi-Modal Models

• URL: http://arxiv.org/abs/2505.05635v1
• Date: Thu, 08 May 2025 20:33:31 GMT
• Title: VR-RAG: Open-vocabulary Species Recognition with RAG-Assisted Large Multi-Modal Models
• Abstract summary: We focus on open-vocabulary bird species recognition, where the goal is to classify species based on their descriptions.<n>Traditional benchmarks like CUB-200-2011 have been evaluated in a closed-vocabulary paradigm.<n>We show that the performance of current systems when evaluated under settings closely aligned with open-vocabulary drops by a huge margin.
• Score: 33.346206174676794
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-vocabulary recognition remains a challenging problem in computer vision, as it requires identifying objects from an unbounded set of categories. This is particularly relevant in nature, where new species are discovered every year. In this work, we focus on open-vocabulary bird species recognition, where the goal is to classify species based on their descriptions without being constrained to a predefined set of taxonomic categories. Traditional benchmarks like CUB-200-2011 and Birdsnap have been evaluated in a closed-vocabulary paradigm, limiting their applicability to real-world scenarios where novel species continually emerge. We show that the performance of current systems when evaluated under settings closely aligned with open-vocabulary drops by a huge margin. To address this gap, we propose a scalable framework integrating structured textual knowledge from Wikipedia articles of 11,202 bird species distilled via GPT-4o into concise, discriminative summaries. We propose Visual Re-ranking Retrieval-Augmented Generation(VR-RAG), a novel, retrieval-augmented generation framework that uses visual similarities to rerank the top m candidates retrieved by a set of multimodal vision language encoders. This allows for the recognition of unseen taxa. Extensive experiments across five established classification benchmarks show that our approach is highly effective. By integrating VR-RAG, we improve the average performance of state-of-the-art Large Multi-Modal Model QWEN2.5-VL by 15.4% across five benchmarks. Our approach outperforms conventional VLM-based approaches, which struggle with unseen species. By bridging the gap between encyclopedic knowledge and visual recognition, our work advances open-vocabulary recognition, offering a flexible, scalable solution for biodiversity monitoring and ecological research.

Related papers

• Coarse-to-fine Hierarchical Architecture with Sequential Mamba for Brain Reconstruction [5.732406989501624]
  We propose CHASMBrain, a novel hierarchical two-stage framework for image-to-fMRI encoding.<n>Our architecture leverages a dual-stream Mamba design to explicitly separate and process global semantic tokens and local spatial patches.<n>Our method achieves a Pearson correlation of 0.429 and an MSE of 0.261, outperforming all evaluated baselines.
  arXiv  Detail & Related papers  (2026-06-03T11:53:23Z)
• ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation [23.118080583803266]
  We present ROVI, a high-quality synthetic dataset for instance-grounded text-to-image generation.<n>Our key innovation is a strategy called recaptioning, focusing on the pre-detection stage.<n>For demonstrative purposes, a text-to-image model GLIGEN trained on ROVI significantly outperforms state-of-the-art alternatives in instance grounding accuracy, prompt fidelity, and aesthetic quality.
  arXiv  Detail & Related papers  (2025-08-01T18:19:51Z)
• Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model [52.01031460230826]
  Traditional approaches rely heavily on fixed vocabularies and closed-set classification paradigms.<n>Recent research has demonstrated that combining large language models with vision-language models (VLMs) makes open-set recognition possible.<n>We propose our training-free method, Enriched-FineR, which demonstrates state-of-the-art results in fine-grained visual recognition.
  arXiv  Detail & Related papers  (2025-07-30T20:06:01Z)
• Multi-scale Activation, Refinement, and Aggregation: Exploring Diverse Cues for Fine-Grained Bird Recognition [35.99227153038734]
  Fine-Grained Bird Recognition (FGBR) has gained increasing attention.<n>Recent studies reveal that the limited receptive field of plain ViT model hinders representational richness.<n>We propose a novel framework for FGBR, namely Multi-scale Diverse Cues Modeling (MDCM)
  arXiv  Detail & Related papers  (2025-04-12T13:47:24Z)
• Taxonomic Reasoning for Rare Arthropods: Combining Dense Image Captioning and RAG for Interpretable Classification [12.923336716880506]
  We integrate image captioning and retrieval-augmented generation (RAG) with large language models (LLMs) to enhance biodiversity monitoring.<n>Our findings highlight the potential for modern vision-language AI pipelines to support biodiversity conservation initiatives.
  arXiv  Detail & Related papers  (2025-03-13T21:18:10Z)
• Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs [65.93003087656754]
  VisFactor is a benchmark that digitizes 20 vision-centric subtests from a well-established cognitive psychology assessment.<n>We evaluate 20 frontier Multimodal Large Language Models (MLLMs) from GPT, Gemini, Claude, LLaMA, Qwen, and SEED families.<n>The best-performing model achieves a score of only 25.19 out of 100, with consistent failures on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination.
  arXiv  Detail & Related papers  (2025-02-23T04:21:32Z)
• Beyond Coarse-Grained Matching in Video-Text Retrieval [50.799697216533914]
  We introduce a new approach for fine-grained evaluation. Our approach can be applied to existing datasets by automatically generating hard negative test captions. Experiments on our fine-grained evaluations demonstrate that this approach enhances a model's ability to understand fine-grained differences.
  arXiv  Detail & Related papers  (2024-10-16T09:42:29Z)
• LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction [63.668635390907575]
  Existing methods enhance open-vocabulary object detection by leveraging the robust open-vocabulary recognition capabilities of Vision-Language Models (VLMs) We propose the Language Model Instruction (LaMI) strategy, which leverages the relationships between visual concepts and applies them within a simple yet effective DETR-like detector.
  arXiv  Detail & Related papers  (2024-07-16T02:58:33Z)
• Dual-Image Enhanced CLIP for Zero-Shot Anomaly Detection [58.228940066769596]
  We introduce a Dual-Image Enhanced CLIP approach, leveraging a joint vision-language scoring system. Our methods process pairs of images, utilizing each as a visual reference for the other, thereby enriching the inference process with visual context. Our approach significantly exploits the potential of vision-language joint anomaly detection and demonstrates comparable performance with current SOTA methods across various datasets.
  arXiv  Detail & Related papers  (2024-05-08T03:13:20Z)
• DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition [28.02038637078298]
  Open-Vocabulary Action Recognition (OVAR) is one of the fundamental video tasks in computer vision. This paper pioneers to evaluate existing methods by simulating multi-level noises of various types. We propose one novel DENOISER framework, covering two parts: generation and discrimination.
  arXiv  Detail & Related papers  (2024-04-23T10:17:42Z)
• Evolving Interpretable Visual Classifiers with Large Language Models [34.4903887876357]
  Multimodal pre-trained models, such as CLIP, are popular for zero-shot classification due to their open-vocabulary flexibility and high performance. vision-language models, which compute similarity scores between images and class labels, are largely black-box, with limited interpretability, risk for bias, and inability to discover new visual concepts not written down. We present a novel method that discovers interpretable yet discriminative sets of attributes for visual recognition.
  arXiv  Detail & Related papers  (2024-04-15T17:09:53Z)
• ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognition [5.253376886484742]
  We present a vision-language model which employs multi-modal decoding of visual features extracted directly from camera trap videos. We evaluate our system on the PanAf500 and PanAf20K datasets. We achieve state-of-the-art performance over vision and vision-language models in top-1 accuracy.
  arXiv  Detail & Related papers  (2024-04-13T09:17:51Z)
• RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition [78.97487780589574]
  Multimodal Large Language Models (MLLMs) excel at classifying fine-grained categories. This paper introduces a Retrieving And Ranking augmented method for MLLMs. Our proposed approach not only addresses the inherent limitations in fine-grained recognition but also preserves the model's comprehensive knowledge base.
  arXiv  Detail & Related papers  (2024-03-20T17:59:55Z)
• Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching [74.75284453828017]
  Open-Vocabulary Keypoint Detection (OVKD) task is innovatively designed to use text prompts for identifying arbitrary keypoints across any species. We have developed a novel framework named Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM) This framework combines vision and language models, creating an interplay between language features and local keypoint visual features.
  arXiv  Detail & Related papers  (2023-10-08T07:42:41Z)
• Distillation-guided Representation Learning for Unconstrained Gait Recognition [50.0533243584942]
  We propose a framework, termed GAit DEtection and Recognition (GADER), for human authentication in challenging outdoor scenarios. GADER builds discriminative features through a novel gait recognition method, where only frames containing gait information are used. We evaluate our method on multiple State-of-The-Arts(SoTA) gait baselines and demonstrate consistent improvements on indoor and outdoor datasets.
  arXiv  Detail & Related papers  (2023-07-27T01:53:57Z)
• Multi-View Active Fine-Grained Recognition [29.980409725777292]
  Fine-grained visual classification (FGVC) is being developed for decades. Discriminative information is not only present within seen local regions but also hides in other unseen perspectives. We propose a policy-gradient-based framework to achieve efficient recognition with active view selection.
  arXiv  Detail & Related papers  (2022-06-02T17:12:14Z)
• Spatio-temporal Relation Modeling for Few-shot Action Recognition [100.3999454780478]
  We propose a few-shot action recognition framework, STRM, which enhances class-specific featureriminability while simultaneously learning higher-order temporal representations. Our approach achieves an absolute gain of 3.5% in classification accuracy, as compared to the best existing method in the literature.
  arXiv  Detail & Related papers  (2021-12-09T18:59:14Z)
• Distribution Alignment: A Unified Framework for Long-tail Visual Recognition [52.36728157779307]
  We propose a unified distribution alignment strategy for long-tail visual recognition. We then introduce a generalized re-weight method in the two-stage learning to balance the class prior. Our approach achieves the state-of-the-art results across all four recognition tasks with a simple and unified framework.
  arXiv  Detail & Related papers  (2021-03-30T14:09:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.