dacl1k: Real-World Bridge Damage Dataset Putting Open-Source Data to the
Test
- URL: http://arxiv.org/abs/2309.03763v1
- Date: Thu, 7 Sep 2023 15:05:35 GMT
- Title: dacl1k: Real-World Bridge Damage Dataset Putting Open-Source Data to the
Test
- Authors: Johannes Flotzinger, Philipp J. R\"osch, Norbert Oswald, Thomas Braml
- Abstract summary: "dacl1k" is a multi-label RCD dataset for multi-label classification based on building inspections including 1,474 images.
We trained the models on different combinations of open-source data (meta datasets) which were subsequently evaluated both extrinsically and intrinsically.
The performance analysis on dacl1k shows practical usability of the meta data, where the best model shows an Exact Match Ratio of 32%.
- Score: 0.6827423171182154
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recognising reinforced concrete defects (RCDs) is a crucial element for
determining the structural integrity, traffic safety and durability of bridges.
However, most of the existing datasets in the RCD domain are derived from a
small number of bridges acquired in specific camera poses, lighting conditions
and with fixed hardware. These limitations question the usability of models
trained on such open-source data in real-world scenarios. We address this
problem by testing such models on our "dacl1k" dataset, a highly diverse RCD
dataset for multi-label classification based on building inspections including
1,474 images. Thereby, we trained the models on different combinations of
open-source data (meta datasets) which were subsequently evaluated both
extrinsically and intrinsically. During extrinsic evaluation, we report metrics
on dacl1k and the meta datasets. The performance analysis on dacl1k shows
practical usability of the meta data, where the best model shows an Exact Match
Ratio of 32%. Additionally, we conduct an intrinsic evaluation by clustering
the bottleneck features of the best model derived from the extrinsic evaluation
in order to find out, if the model has learned distinguishing datasets or the
classes (RCDs) which is the aspired goal. The dacl1k dataset and our trained
models will be made publicly available, enabling researchers and practitioners
to put their models to the real-world test.
Related papers
- SARDet-100K: Towards Open-Source Benchmark and ToolKit for Large-Scale
SAR Object Detection [83.21028626585986]
We establish a new benchmark dataset and an open-source method for large-scale SAR object detection.
Our dataset, SARDet-100K, is a result of intense surveying, collecting, and standardizing 10 existing SAR detection datasets.
To the best of our knowledge, SARDet-100K is the first COCO-level large-scale multi-class SAR object detection dataset ever created.
arXiv Detail & Related papers (2024-03-11T09:20:40Z) - dacl10k: Benchmark for Semantic Bridge Damage Segmentation [0.0]
"dacl10k" is an exceptionally diverse RCD dataset for semantic segmentation comprising 9,920 images deriving from real-world bridge inspections.
"dacl10k" distinguishes 12 damage classes as well as 6 bridge components that play a key role in the building assessment and recommending actions.
arXiv Detail & Related papers (2023-09-01T13:46:24Z) - Zero-shot Composed Text-Image Retrieval [72.43790281036584]
We consider the problem of composed image retrieval (CIR)
It aims to train a model that can fuse multi-modal information, e.g., text and images, to accurately retrieve images that match the query, extending the user's expression ability.
arXiv Detail & Related papers (2023-06-12T17:56:01Z) - Preserving Knowledge Invariance: Rethinking Robustness Evaluation of
Open Information Extraction [50.62245481416744]
We present the first benchmark that simulates the evaluation of open information extraction models in the real world.
We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.
By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
arXiv Detail & Related papers (2023-05-23T12:05:09Z) - Revisiting Table Detection Datasets for Visually Rich Documents [17.846536373106268]
This study revisits some open datasets with high-quality annotations, identifies and cleans the noise, and aligns the annotation definitions of these datasets to merge a larger dataset, termed Open-Tables.
To enrich the data sources, we propose a new ICT-TD dataset using the PDF files of Information and Communication Technologies (ICT) commodities, a different domain containing unique samples that hardly appear in open datasets.
Our experimental results show that the domain differences among existing open datasets are minor despite having different data sources.
arXiv Detail & Related papers (2023-05-04T01:08:15Z) - Scene-centric vs. Object-centric Image-Text Cross-modal Retrieval: A
Reproducibility Study [55.964387734180114]
Cross-modal retrieval (CMR) approaches usually focus on object-centric datasets.
This paper focuses on the results and their generalizability across different dataset types.
We select two state-of-the-art CMR models with different architectures.
We determine the relative performance of the selected models on these datasets.
arXiv Detail & Related papers (2023-01-12T18:00:00Z) - MSeg: A Composite Dataset for Multi-domain Semantic Segmentation [100.17755160696939]
We present MSeg, a composite dataset that unifies semantic segmentation datasets from different domains.
We reconcile the generalization and bring the pixel-level annotations into alignment by relabeling more than 220,000 object masks in more than 80,000 images.
A model trained on MSeg ranks first on the WildDash-v1 leaderboard for robust semantic segmentation, with no exposure to WildDash data during training.
arXiv Detail & Related papers (2021-12-27T16:16:35Z) - Label-Free Model Evaluation with Semi-Structured Dataset Representations [78.54590197704088]
Label-free model evaluation, or AutoEval, estimates model accuracy on unlabeled test sets.
In the absence of image labels, based on dataset representations, we estimate model performance for AutoEval with regression.
We propose a new semi-structured dataset representation that is manageable for regression learning while containing rich information for AutoEval.
arXiv Detail & Related papers (2021-12-01T18:15:58Z) - BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information
Retrieval Models [41.45240621979654]
We introduce BEIR, a heterogeneous benchmark for information retrieval.
We study the effectiveness of nine state-of-the-art retrieval models in a zero-shot evaluation setup.
Dense-retrieval models are computationally more efficient but often underperform other approaches.
arXiv Detail & Related papers (2021-04-17T23:29:55Z) - On the Composition and Limitations of Publicly Available COVID-19 X-Ray
Imaging Datasets [0.0]
Data scarcity, mismatch between training and target population, group imbalance, and lack of documentation are important sources of bias.
This paper presents an overview of the currently public available COVID-19 chest X-ray datasets.
arXiv Detail & Related papers (2020-08-26T14:16:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.