The Effect of Ground Truth Accuracy on the Evaluation of Localization Systems

• URL: http://arxiv.org/abs/2106.13614v1
• Date: Fri, 25 Jun 2021 13:07:59 GMT
• Title: The Effect of Ground Truth Accuracy on the Evaluation of Localization Systems
• Authors: Chen Gu, Ahmed Shokry, Moustafa Youssef
• Abstract summary: We present a theoretical framework for analyzing the effect of ground truth errors on the evaluation of localization systems. Based on that, we design two algorithms for computing the real algorithmic error from the validation error and marking/map ground truth errors. Our marking error algorithm matches the real error CDF within 4%, and our map error algorithm provides a more accurate estimate of the median/tail error by 150%/72% when the map is shifted by 6m.
• Score: 6.702815463331469
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability to accurately evaluate the performance of location determination systems is crucial for many applications. Typically, the performance of such systems is obtained by comparing ground truth locations with estimated locations. However, these ground truth locations are usually obtained by clicking on a map or using other worldwide available technologies like GPS. This introduces ground truth errors that are due to the marking process, map distortions, or inherent GPS inaccuracy. In this paper, we present a theoretical framework for analyzing the effect of ground truth errors on the evaluation of localization systems. Based on that, we design two algorithms for computing the real algorithmic error from the validation error and marking/map ground truth errors, respectively. We further establish bounds on different performance metrics. Validation of our theoretical assumptions and analysis using real data collected in a typical environment shows the ability of our theoretical framework to correct the estimated error of a localization algorithm in the presence of ground truth errors. Specifically, our marking error algorithm matches the real error CDF within 4%, and our map error algorithm provides a more accurate estimate of the median/tail error by 150%/72% when the map is shifted by 6m.

Related papers

• Detecting Errors in a Numerical Response via any Regression Model [21.651775224356214]
  Noise plagues many numerical datasets, where the recorded values in the data may fail to match the true underlying values. We introduce veracity scores that distinguish between genuine errors and natural data fluctuations. We also contribute a new error detection benchmark involving 5 regression datasets with real-world numerical errors.
  arXiv  Detail & Related papers  (2023-05-26T02:15:26Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• Learnable Spatio-Temporal Map Embeddings for Deep Inertial Localization [21.300194809454077]
  We propose a data-driven prior on possible user locations in a map. Our prior learns to encode which map regions are feasible locations for a user more accurately than previous hand-defined methods.
  arXiv  Detail & Related papers  (2022-11-14T18:58:21Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles [38.23896575179384]
  We propose a principled and practically effective framework that simultaneously addresses the two tasks. One instantiation reduces the estimation error for unsupervised accuracy estimation by at least 70% and improves the F1 score for error detection by at least 4.7%. On iWildCam, one instantiation reduces the estimation error for unsupervised accuracy estimation by at least 70% and improves the F1 score for error detection by at least 4.7%.
  arXiv  Detail & Related papers  (2021-06-29T21:32:51Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Deep Multi-Task Learning for Joint Localization, Perception, and Prediction [68.50217234419922]
  This paper investigates the issues that arise in state-of-the-art autonomy stacks under localization error. We design a system that jointly performs perception, prediction, and localization. Our architecture is able to reuse computation between both tasks, and is thus able to correct localization errors efficiently.
  arXiv  Detail & Related papers  (2021-01-17T17:20:31Z)
• Deep Learning for Earth Image Segmentation based on Imperfect Polyline Labels with Annotation Errors [12.547819302858045]
  This paper proposes a generic learning framework based on the EM algorithm to update deep learning model parameters and infer hidden true label locations simultaneously. Evaluations on a real-world hydrological dataset in the streamline refinement application show that the proposed framework outperforms baseline methods in classification accuracy.
  arXiv  Detail & Related papers  (2020-10-02T02:54:06Z)
• Heatmap Regression via Randomized Rounding [105.75014893647538]
  We propose a simple yet effective quantization system to address the sub-pixel localization problem. The proposed system encodes the fractional part of numerical coordinates into the ground truth heatmap using a probabilistic approach during training.
  arXiv  Detail & Related papers  (2020-09-01T04:54:22Z)
• Rethinking Localization Map: Towards Accurate Object Perception with Self-Enhancement Maps [78.2581910688094]
  This work introduces a novel self-enhancement method to harvest accurate object localization maps and object boundaries with only category labels as supervision. In particular, the proposed Self-Enhancement Maps achieve the state-of-the-art localization accuracy of 54.88% on ILSVRC.
  arXiv  Detail & Related papers  (2020-06-09T12:35:55Z)
• 3D Lidar Mapping Relative Accuracy Automatic Evaluation Algorithm [7.4977759910523964]
  HD (High Definition) map based on 3D lidar plays a vital role in autonomous vehicle localization, planning, decision-making, perception, etc. Many 3D lidar mapping technologies related to SLAM are used in HD map construction to ensure its high accuracy. To evaluate the accuracy of 3D lidar mapping, the most common methods use ground truth of poses to calculate the error between estimated poses and ground truth. In this paper, we proposed a relative accuracy evaluation algorithm that can automatically evaluate the accuracy of HD map built by 3D lidar mapping without ground truth.
  arXiv  Detail & Related papers  (2020-06-01T11:30:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.