The Effectiveness of Edge Detection Evaluation Metrics for Automated Coastline Detection

• URL: http://arxiv.org/abs/2405.11498v1
• Date: Sun, 19 May 2024 09:51:10 GMT
• Title: The Effectiveness of Edge Detection Evaluation Metrics for Automated Coastline Detection
• Authors: Conor O'Sullivan, Seamus Coveney, Xavier Monteys, Soumyabrata Dev,
• Abstract summary: We evaluate RMSE, PSNR, SSIM and FOM for automated coastline detection. We apply Canny edge detection to 95 coastline satellite images across 49 testing locations. FOM was the most reliable metric for selecting the best threshold.
• Score: 2.5311562666866494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We analyse the effectiveness of RMSE, PSNR, SSIM and FOM for evaluating edge detection algorithms used for automated coastline detection. Typically, the accuracy of detected coastlines is assessed visually. This can be impractical on a large scale leading to the need for objective evaluation metrics. Hence, we conduct an experiment to find reliable metrics. We apply Canny edge detection to 95 coastline satellite images across 49 testing locations. We vary the Hysteresis thresholds and compare metric values to a visual analysis of detected edges. We found that FOM was the most reliable metric for selecting the best threshold. It could select a better threshold 92.6% of the time and the best threshold 66.3% of the time. This is compared RMSE, PSNR and SSIM which could select the best threshold 6.3%, 6.3% and 11.6% of the time respectively. We provide a reason for these results by reformulating RMSE, PSNR and SSIM in terms of confusion matrix measures. This suggests these metrics not only fail for this experiment but are not useful for evaluating edge detection in general.

Related papers

• Towards a Trustworthy Anomaly Detection for Critical Applications through Approximated Partial AUC Loss [2.09942566943801]
  A binary classifier is trained to optimize the specific range of the AUC ROC curve that prevents the True Positive Rate (TPR) to reach 100% while minimizing the False Positive Rate (FPR) The results show a TPR of 92.52% at a 20.43% FPR for an average across 6 datasets, representing a TPR improvement of 4.3% for a FPR cost of 12.2% against other state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-02-17T08:59:59Z)
• Model-free Test Time Adaptation for Out-Of-Distribution Detection [62.49795078366206]
  We propose a Non-Parametric Test Time textbfAdaptation framework for textbfDistribution textbfDetection (abbr) abbr utilizes online test samples for model adaptation during testing, enhancing adaptability to changing data distributions. We demonstrate the effectiveness of abbr through comprehensive experiments on multiple OOD detection benchmarks.
  arXiv  Detail & Related papers  (2023-11-28T02:00:47Z)
• Delving into Crispness: Guided Label Refinement for Crisp Edge Detection [14.26122188105415]
  Learning-based edge detection usually suffers from predicting thick edges. We find that noisy human-labeled edges are the main cause of thick predictions. We propose an effective Canny-guided refinement of human-labeled edges.
  arXiv  Detail & Related papers  (2023-06-27T03:12:58Z)
• Beyond AUROC & co. for evaluating out-of-distribution detection performance [50.88341818412508]
  Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. We propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples.
  arXiv  Detail & Related papers  (2023-06-26T12:51:32Z)
• Analysis of ROC for Edge Detectors [14.494626833445915]
  This paper presents an evaluation of edge detectors using receiver operating characteristic (ROC) analysis on the BIPED dataset. We observed that while ROC analysis is suitable for certain edge filters, it presents challenges when accurately measuring their performance using ROC metrics. To address this issue, we introduce customization techniques to enhance the performance of these filters, enabling more accurate evaluation.
  arXiv  Detail & Related papers  (2023-05-28T22:47:54Z)
• An Experimental Investigation into the Evaluation of Explainability Methods [60.54170260771932]
  This work compares 14 different metrics when applied to nine state-of-the-art XAI methods and three dummy methods (e.g., random saliency maps) used as references. Experimental results show which of these metrics produces highly correlated results, indicating potential redundancy.
  arXiv  Detail & Related papers  (2023-05-25T08:07:07Z)
• Out-of-Distribution Detection with Hilbert-Schmidt Independence Optimization [114.43504951058796]
  Outlier detection tasks have been playing a critical role in AI safety. Deep neural network classifiers usually tend to incorrectly classify out-of-distribution (OOD) inputs into in-distribution classes with high confidence. We propose an alternative probabilistic paradigm that is both practically useful and theoretically viable for the OOD detection tasks.
  arXiv  Detail & Related papers  (2022-09-26T15:59:55Z)
• Just-Noticeable-Difference Based Edge Map Quality Measure [5.749044590090683]
  Distance-based edge map measures are widely used for assessment of edge map quality. This paper presents edge map quality measure based on Just-Noticeable-Difference (JND) feature of human visual system.
  arXiv  Detail & Related papers  (2022-04-07T01:34:30Z)
• Android Malware Detection using Feature Ranking of Permissions [0.0]
  We use Android permissions as a vehicle to allow for quick and effective differentiation between benign and malware apps. Our analysis indicates that this approach can result in better accuracy and F-score value than other reported approaches.
  arXiv  Detail & Related papers  (2022-01-20T22:08:20Z)
• Localization Uncertainty-Based Attention for Object Detection [8.154943252001848]
  We propose a more efficient uncertainty-aware dense detector (UADET) that predicts four-directional localization uncertainties via Gaussian modeling. Experiments using the MS COCO benchmark show that our UADET consistently surpasses baseline FCOS, and that our best model, ResNext-64x4d-101-DCN, obtains a single model, single-scale AP of 48.3% on COCO test-dev.
  arXiv  Detail & Related papers  (2021-08-25T04:32:39Z)
• A Novel Approach to Radiometric Identification [68.8204255655161]
  This paper demonstrates that highly accurate radiometric identification is possible using CAPoNeF feature engineering method. We tested basic ML classification algorithms on experimental data gathered by SDR.
  arXiv  Detail & Related papers  (2020-12-02T10:54:44Z)
• Bayesian Optimization with Machine Learning Algorithms Towards Anomaly Detection [66.05992706105224]
  In this paper, an effective anomaly detection framework is proposed utilizing Bayesian Optimization technique. The performance of the considered algorithms is evaluated using the ISCX 2012 dataset. Experimental results show the effectiveness of the proposed framework in term of accuracy rate, precision, low-false alarm rate, and recall.
  arXiv  Detail & Related papers  (2020-08-05T19:29:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.