Effective Benchmarks for Optical Turbulence Modeling
- URL: http://arxiv.org/abs/2401.03573v1
- Date: Sun, 7 Jan 2024 20:00:35 GMT
- Title: Effective Benchmarks for Optical Turbulence Modeling
- Authors: Christopher Jellen and Charles Nelson and Cody Brownell and John
Burkhardt
- Abstract summary: We introduce the textttotbench package, a Python package for rigorous development and evaluation of optical turbulence strength prediction models.
The package provides a consistent interface for evaluating optical turbulence models on a variety of benchmark tasks and data sets.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Optical turbulence presents a significant challenge for communication,
directed energy, and imaging systems, especially in the atmospheric boundary
layer. Effective modeling of optical turbulence strength is critical for the
development and deployment of these systems. The lack of standard evaluation
tools, especially long-term data sets, modeling tasks, metrics, and baseline
models, prevent effective comparisons between approaches and models. This
reduces the ease of reproducing results and contributes to over-fitting on
local micro-climates. Performance characterized using evaluation metrics
provides some insight into the applicability of a model for predicting the
strength of optical turbulence. However, these metrics are not sufficient for
understanding the relative quality of a model. We introduce the
\texttt{otbench} package, a Python package for rigorous development and
evaluation of optical turbulence strength prediction models. The package
provides a consistent interface for evaluating optical turbulence models on a
variety of benchmark tasks and data sets. The \texttt{otbench} package includes
a range of baseline models, including statistical, data-driven, and deep
learning models, to provide a sense of relative model quality. \texttt{otbench}
also provides support for adding new data sets, tasks, and evaluation metrics.
The package is available at \url{https://github.com/cdjellen/otbench}.
Related papers
- Meta-learning and Data Augmentation for Stress Testing Forecasting Models [0.33554367023486936]
A model is considered to be under stress if it shows a negative behaviour, such as higher-than-usual errors or increased uncertainty.
This paper contributes with a novel framework called MAST (Meta-learning and data Augmentation for Stress Testing)
arXiv Detail & Related papers (2024-06-24T17:59:33Z) - Data-efficient Large Vision Models through Sequential Autoregression [58.26179273091461]
We develop an efficient, autoregression-based vision model on a limited dataset.
We demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding.
Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint.
arXiv Detail & Related papers (2024-02-07T13:41:53Z) - QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement.
QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights.
We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
arXiv Detail & Related papers (2023-11-06T00:21:44Z) - Robustness and Generalization Performance of Deep Learning Models on
Cyber-Physical Systems: A Comparative Study [71.84852429039881]
Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise.
We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
arXiv Detail & Related papers (2023-06-13T12:43:59Z) - Studying How to Efficiently and Effectively Guide Models with Explanations [52.498055901649025]
'Model guidance' is the idea of regularizing the models' explanations to ensure that they are "right for the right reasons"
We conduct an in-depth evaluation across various loss functions, attribution methods, models, and 'guidance depths' on the PASCAL VOC 2007 and MS COCO 2014 datasets.
Specifically, we guide the models via bounding box annotations, which are much cheaper to obtain than the commonly used segmentation masks.
arXiv Detail & Related papers (2023-03-21T15:34:50Z) - Information-Theoretic Odometry Learning [83.36195426897768]
We propose a unified information theoretic framework for learning-motivated methods aimed at odometry estimation.
The proposed framework provides an elegant tool for performance evaluation and understanding in information-theoretic language.
arXiv Detail & Related papers (2022-03-11T02:37:35Z) - Distributional Depth-Based Estimation of Object Articulation Models [21.046351215949525]
We propose a method that efficiently learns distributions over articulation model parameters directly from depth images.
Our core contributions include a novel representation for distributions over rigid body transformations.
We introduce a novel deep learning based approach, DUST-net, that performs category-independent articulation model estimation.
arXiv Detail & Related papers (2021-08-12T17:44:51Z) - SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for
Text Summarization [14.787106201073154]
SummVis is an open-source tool for visualizing abstractive summaries.
It enables fine-grained analysis of the models, data, and evaluation metrics associated with text summarization.
arXiv Detail & Related papers (2021-04-15T17:13:00Z) - How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating
and Auditing Generative Models [95.8037674226622]
We introduce a 3-dimensional evaluation metric that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion.
Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity.
arXiv Detail & Related papers (2021-02-17T18:25:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.