Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation

• URL: http://arxiv.org/abs/2412.02262v1
• Date: Tue, 03 Dec 2024 08:34:42 GMT
• Title: Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation
• Authors: Sepand Dyanatkar, Angran Li, Alexander Dungate,
• Abstract summary: We propose a resilient, scalable solution for image and video analysis in marine applications. We validate this approach through a preliminary application in classifying fish from video onboard fishing vessels.
• Score: 41.94295877935867
• License:
• Abstract: Climate change's destruction of marine biodiversity is threatening communities and economies around the world which rely on healthy oceans for their livelihoods. The challenge of applying computer vision to niche, real-world domains such as ocean conservation lies in the dynamic and diverse environments where traditional top-down learning struggle with long-tailed distributions, generalization, and domain transfer. Scalable species identification for ocean monitoring is particularly difficult due to the need to adapt models to new environments and identify rare or unseen species. To overcome these limitations, we propose leveraging bottom-up, open-domain learning frameworks as a resilient, scalable solution for image and video analysis in marine applications. Our preliminary demonstration uses pretrained vision-language models (VLMs) combined with retrieval-augmented generation (RAG) as grounding, leaving the door open for numerous architectural, training and engineering optimizations. We validate this approach through a preliminary application in classifying fish from video onboard fishing vessels, demonstrating impressive emergent retrieval and prediction capabilities without domain-specific training or knowledge of the task itself.

Related papers

• Harnessing Artificial Intelligence for Wildlife Conservation [0.0937465283958018]
  Conservation AI detects and classifies animals, humans, and poaching-related objects using visual spectrum and thermal infrared cameras. The platform processes this data with convolutional neural networks (CNNs) and Transformer architectures to monitor species. Case studies from Europe, North America, Africa, and Southeast Asia highlight the platform's success in species identification, biodiversity monitoring, and poaching prevention.
  arXiv  Detail & Related papers  (2024-08-30T09:13:31Z)
• A Computer Vision Approach to Estimate the Localized Sea State [45.498315114762484]
  This research focuses on utilizing sea images in operational envelopes captured by a single stationary camera mounted on the ship bridge. The collected images are used to train a deep learning model to automatically recognize the state of the sea based on the Beaufort scale.
  arXiv  Detail & Related papers  (2024-07-04T09:07:25Z)
• Multifidelity digital twin for real-time monitoring of structural dynamics in aquaculture net cages [1.124958340749622]
  Digital twin technology can advance the aquaculture industry, but its adoption has been limited. Fish net cages, which are flexible floating structures, are critical yet vulnerable components of aquaculture farms. We propose a multifidelity surrogate modeling framework for integration into a digital twin for real-time monitoring of aquaculture net cage structural dynamics.
  arXiv  Detail & Related papers  (2024-06-06T21:26:30Z)
• A Survey on Transferability of Adversarial Examples across Deep Neural Networks [53.04734042366312]
  adversarial examples can manipulate machine learning models into making erroneous predictions. The transferability of adversarial examples enables black-box attacks which circumvent the need for detailed knowledge of the target model. This survey explores the landscape of the adversarial transferability of adversarial examples.
  arXiv  Detail & Related papers  (2023-10-26T17:45:26Z)
• FisHook -- An Optimized Approach to Marine Specie Classification using MobileNetV2 [5.565562836494568]
  classification and monitoring of marine species can aid in understanding their distribution, population dynamics, and the impact of human activities on them. Deep-learning algorithms can now efficiently classify marine species, making it easier to monitor and manage marine ecosystems.
  arXiv  Detail & Related papers  (2023-04-04T04:30:25Z)
• GeoNet: Benchmarking Unsupervised Adaptation across Geographies [71.23141626803287]
  We study the problem of geographic robustness and make three main contributions. First, we introduce a large-scale dataset GeoNet for geographic adaptation. Second, we hypothesize that the major source of domain shifts arise from significant variations in scene context. Third, we conduct an extensive evaluation of several state-of-the-art unsupervised domain adaptation algorithms and architectures.
  arXiv  Detail & Related papers  (2023-03-27T17:59:34Z)
• Applications of Deep Learning in Fish Habitat Monitoring: A Tutorial and Survey [1.9249287163937976]
  Deep learning (DL) is a cutting-edge AI technology that has demonstrated unprecedented performance in analysing visual data. In this paper, we provide a tutorial that covers the key concepts of DL, which help the reader grasp a high-level understanding of how DL works. The tutorial also explains a step-by-step procedure on how DL algorithms should be developed for challenging applications such as underwater fish monitoring.
  arXiv  Detail & Related papers  (2022-06-11T01:59:54Z)
• Unlocking the potential of deep learning for marine ecology: overview, applications, and outlook [8.3226670069051]
  This paper aims to bridge the gap between marine ecologists and computer scientists. We provide insight into popular deep learning approaches for ecological data analysis in plain language. We illustrate challenges and opportunities through established and emerging applications of deep learning to marine ecology.
  arXiv  Detail & Related papers  (2021-09-29T21:59:16Z)
• Movement Tracks for the Automatic Detection of Fish Behavior in Videos [63.85815474157357]
  We offer a dataset of sablefish (Anoplopoma fimbria) startle behaviors in underwater videos, and investigate the use of deep learning (DL) methods for behavior detection on it. Our proposed detection system identifies fish instances using DL-based frameworks, determines trajectory tracks, derives novel behavior-specific features, and employs Long Short-Term Memory (LSTM) networks to identify startle behavior in sablefish.
  arXiv  Detail & Related papers  (2020-11-28T05:51:19Z)
• Physics-informed GANs for Coastal Flood Visualization [65.54626149826066]
  We create a deep learning pipeline that generates visual satellite images of current and future coastal flooding. By evaluating the imagery relative to physics-based flood maps, we find that our proposed framework outperforms baseline models in both physical-consistency and photorealism. While this work focused on the visualization of coastal floods, we envision the creation of a global visualization of how climate change will shape our earth.
  arXiv  Detail & Related papers  (2020-10-16T02:15:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.