GeoBench: Benchmarking and Analyzing Monocular Geometry Estimation Models

• URL: http://arxiv.org/abs/2406.12671v2
• Date: Fri, 21 Jun 2024 02:45:10 GMT
• Title: GeoBench: Benchmarking and Analyzing Monocular Geometry Estimation Models
• Authors: Yongtao Ge, Guangkai Xu, Zhiyue Zhao, Libo Sun, Zheng Huang, Yanlong Sun, Hao Chen, Chunhua Shen,
• Abstract summary: Discriminative and generative pretraining have yielded geometry estimation models with strong generalization capabilities. We build fair and strong baselines for evaluating and analyzing the geometry estimation models. We evaluate monocular geometry estimators on more challenging benchmarks for geometry estimation task with diverse scenes and high-quality annotations.
• Score: 41.76935689355034
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in discriminative and generative pretraining have yielded geometry estimation models with strong generalization capabilities. While discriminative monocular geometry estimation methods rely on large-scale fine-tuning data to achieve zero-shot generalization, several generative-based paradigms show the potential of achieving impressive generalization performance on unseen scenes by leveraging pre-trained diffusion models and fine-tuning on even a small scale of synthetic training data. Frustratingly, these models are trained with different recipes on different datasets, making it hard to find out the critical factors that determine the evaluation performance. Besides, current geometry evaluation benchmarks have two main drawbacks that may prevent the development of the field, i.e., limited scene diversity and unfavorable label quality. To resolve the above issues, (1) we build fair and strong baselines in a unified codebase for evaluating and analyzing the geometry estimation models; (2) we evaluate monocular geometry estimators on more challenging benchmarks for geometry estimation task with diverse scenes and high-quality annotations. Our results reveal that pre-trained using large data, discriminative models such as DINOv2, can outperform generative counterparts with a small amount of high-quality synthetic data under the same training configuration, which suggests that fine-tuning data quality is a more important factor than the data scale and model architecture. Our observation also raises a question: if simply fine-tuning a general vision model such as DINOv2 using a small amount of synthetic depth data produces SOTA results, do we really need complex generative models for depth estimation? We believe this work can propel advancements in geometry estimation tasks as well as a wide range of downstream applications.

Related papers

• FRoundation: Are Foundation Models Ready for Face Recognition? [8.045296450065019]
  We propose and demonstrate the adaptation of foundation models for face recognition across different levels of data availability. Our results indicate that, despite their versatility, pre-trained foundation models underperform in face recognition. Fine-tuning foundation models yields promising results, often surpassing models trained from scratch when training data is limited.
  arXiv  Detail & Related papers  (2024-10-31T11:21:21Z)
• (Deep) Generative Geodesics [57.635187092922976]
  We introduce a newian metric to assess the similarity between any two data points. Our metric leads to the conceptual definition of generative distances and generative geodesics. Their approximations are proven to converge to their true values under mild conditions.
  arXiv  Detail & Related papers  (2024-07-15T21:14:02Z)
• GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models [56.63218531256961]
  We introduce GenBench, a benchmarking suite specifically tailored for evaluating the efficacy of Genomic Foundation Models. GenBench offers a modular and expandable framework that encapsulates a variety of state-of-the-art methodologies. We provide a nuanced analysis of the interplay between model architecture and dataset characteristics on task-specific performance.
  arXiv  Detail & Related papers  (2024-06-01T08:01:05Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Topological Parallax: A Geometric Specification for Deep Perception Models [0.778001492222129]
  We introduce topological parallax as a theoretical and computational tool that compares a trained model to a reference dataset. Our examples show that this geometric similarity between dataset and model is essential to trustworthy and perturbation. This new concept will add value to the current debate regarding the unclear relationship between overfitting and generalization in applications of deep-learning.
  arXiv  Detail & Related papers  (2023-06-20T18:45:24Z)
• Evaluation of Categorical Generative Models -- Bridging the Gap Between Real and Synthetic Data [18.142397311464343]
  We introduce an appropriately scalable evaluation method for generative models. We consider increasingly large probability spaces, which correspond to increasingly difficult modeling tasks. We validate our evaluation procedure with synthetic experiments on both synthetic generative models and current state-of-the-art categorical generative models.
  arXiv  Detail & Related papers  (2022-10-28T21:05:25Z)
• "Understanding Robustness Lottery": A Geometric Visual Comparative Analysis of Neural Network Pruning Approaches [29.048660060344574]
  This work aims to shed light on how different pruning methods alter the network's internal feature representation and the corresponding impact on model performance. We introduce a visual geometric analysis of feature representations to compare and highlight the impact of pruning on model performance and feature representation. The proposed tool provides an environment for in-depth comparison of pruning methods and a comprehensive understanding of how model response to common data corruption.
  arXiv  Detail & Related papers  (2022-06-16T04:44:13Z)
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models [95.8037674226622]
  We introduce a 3-dimensional evaluation metric that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity.
  arXiv  Detail & Related papers  (2021-02-17T18:25:30Z)
• Predicting Multidimensional Data via Tensor Learning [0.0]
  We develop a model that retains the intrinsic multidimensional structure of the dataset. To estimate the model parameters, an Alternating Least Squares algorithm is developed. The proposed model is able to outperform benchmark models present in the forecasting literature.
  arXiv  Detail & Related papers  (2020-02-11T11:57:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.