RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs

• URL: http://arxiv.org/abs/2603.27033v1
• Date: Fri, 27 Mar 2026 22:58:31 GMT
• Title: RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs
• Abstract summary: Fine-grained bird species identification in the wild is frequently unanswerable from a single image.<n>We propose the RealBirdID benchmark: given an image of a bird, a system should either answer with a species or abstain with a concrete, evidence-based rationale.<n>For each genus, the dataset includes a validation split composed of curated unanswerable examples with labeled rationales, paired with a companion set of clearly answerable instances.
• Score: 21.24879709845471
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-grained bird species identification in the wild is frequently unanswerable from a single image: key cues may be non-visual (e.g. vocalization), or obscured due to occlusion, camera angle, or low resolution. Yet today's multimodal systems are typically judged on answerable, in-schema cases, encouraging confident guesses rather than principled abstention. We propose the RealBirdID benchmark: given an image of a bird, a system should either answer with a species or abstain with a concrete, evidence-based rationale: "requires vocalization," "low quality image," or "view obstructed". For each genus, the dataset includes a validation split composed of curated unanswerable examples with labeled rationales, paired with a companion set of clearly answerable instances. We find that (1) the species identification on the answerable set is challenging for a variety of open-source and proprietary models (less than 13% accuracy for MLLMs including GPT-5 and Gemini-2.5 Pro), (2) models with greater classification ability are not necessarily more calibrated to abstain from unanswerable examples, and (3) that MLLMs generally fail at providing correct reasons even when they do abstain. RealBirdID establishes a focused target for abstention-aware fine-grained recognition and a recipe for measuring progress.

Related papers

• Open-Set Cattle Muzzle Identification: A Leakage-Controlled Benchmark and Evaluation Protocol [0.0]
  We reformulate cattle muzzle biometrics as an open-set, gallery-based identification problem.<n>We introduce a leakage-controlled evaluation protocol based on identity-disjoint splits, per-fold retraining, held-out threshold calibration, verified duplicate removal, and bootstrap confidence intervals.
  arXiv  Detail & Related papers  (2026-08-22T15:48:21Z)
• DS@GT ARC at AnimalCLEF 2026: Species-Aware Graph Construction for Multi-Species Animal Re-Identification [0.0]
  The DS@GT ARC submission to AnimalCLEF 2026 introduces a multi-species image-clustering system for re-identification.<n>Instead of relying on a single descriptor or nearest-neighbor retrieval, this approach formulates re-identification as species-aware graph construction.
  arXiv  Detail & Related papers  (2026-07-17T18:49:49Z)
• DNA: Dual-stage Native Attribution for Generated Image Source Tracing [75.65470221927096]
  Existing proactive methods rely on watermark embedding or model modification.<n>DNA is a coarse-to-fine framework that follows this hierarchy without additional neural-network training.<n>Experiments show that DNA achieves 89.11% end-to-end attribution accuracy.
  arXiv  Detail & Related papers  (2026-07-15T10:27:52Z)
• FIKA-Bench: From Fine-grained Recognition to Fine-Grained Knowledge Acquisition [54.31138496553705]
  Fine-grained recognition in everyday life is often not a closed-book classification problem.<n>Existing benchmarks primarily evaluate visually recognition, leaving this active external knowledge acquisition ability underexplored.<n>We study fine-grained knowledge acquisition, where a system must seek, verify, and use external evidence to answer open-ended fine-grained recognition questions.
  arXiv  Detail & Related papers  (2026-05-13T08:49:51Z)
• Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems [52.83669998269706]
  Abstention has been studied in text-only settings but remains underexplored multimodally.<n>Current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes.<n>We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes.
  arXiv  Detail & Related papers  (2026-04-16T09:23:22Z)
• Self-Supervised Animal Identification for Long Videos [0.8233028449337972]
  We introduce a highly efficient, self-supervised method that reframes animal identification as a global clustering task.<n>Our framework matches or surpasses supervised baselines trained on over 1,000 labeled frames.<n>This work enables practical, high-accuracy animal identification on consumer-grade hardware.
  arXiv  Detail & Related papers  (2026-01-14T17:53:59Z)
• Measuring Epistemic Humility in Multimodal Large Language Models [17.490955813494693]
  We present HumbleBench, a new hallucination benchmark designed to evaluate MLLMs' ability to reject plausible but incorrect answers.<n>We leverage fine-grained scene graph annotations to extract ground-truth entities and relations, and prompt GPT-4-Turbo to generate multiple-choice questions.<n>HumbleBench fills a key gap in current evaluation suites, providing a more realistic measure of MLLM reliability in safety-critical settings.
  arXiv  Detail & Related papers  (2025-09-11T17:54:00Z)
• COIN: Confidence Score-Guided Distillation for Annotation-Free Cell Segmentation [2.5234274237739402]
  We present COIN (COnfidence score-guided INstance distillation), a novel annotation-free framework with three key steps.<n>COIN increases sensitivity for the presence of error-free instances via unsupervised semantic segmentation with optimal transport.<n>It offers an alternative to ground truth annotations, offering an alternative to ground truth annotations.
  arXiv  Detail & Related papers  (2025-03-14T14:27:24Z)
• Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness [106.52630978891054]
  We present a taxonomy of uncertainty specific to vision-language AI systems. We also introduce a new metric confidence-weighted accuracy, that is well correlated with both accuracy and calibration error.
  arXiv  Detail & Related papers  (2024-07-02T04:23:54Z)
• Visually Consistent Hierarchical Image Classification [37.80849457554078]
  Hierarchical classification predicts labels across multiple levels of a taxonomy, e.g., from coarse-level 'Bird' to mid-level 'Hummingbird' to fine-level 'Green hermit'
  arXiv  Detail & Related papers  (2024-06-17T14:56:51Z)
• Towards Automated Animal Density Estimation with Acoustic Spatial Capture-Recapture [2.5193666094305938]
  Digital recorders allow surveyors to gather large volumes of data at low cost. But identifying target species vocalisations in these data is non-trivial. Machine learning (ML) methods are often used to do the identification. We propose three methods for acoustic spatial capture-recapture inference.
  arXiv  Detail & Related papers  (2023-08-24T15:29:24Z)
• Improving Selective Visual Question Answering by Learning from Your Peers [74.20167944693424]
  Visual Question Answering (VQA) models can have difficulties abstaining from answering when they are wrong. We propose Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model.
  arXiv  Detail & Related papers  (2023-06-14T21:22:01Z)
• Persistent Animal Identification Leveraging Non-Visual Markers [71.14999745312626]
  We aim to locate and provide a unique identifier for each mouse in a cluttered home-cage environment through time. This is a very challenging problem due to (i) the lack of distinguishing visual features for each mouse, and (ii) the close confines of the scene with constant occlusion. Our approach achieves 77% accuracy on this animal identification problem, and is able to reject spurious detections when the animals are hidden.
  arXiv  Detail & Related papers  (2021-12-13T17:11:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.