Beyond Accuracy: What Matters in Designing Well-Behaved Models?

• URL: http://arxiv.org/abs/2503.17110v1
• Date: Fri, 21 Mar 2025 12:54:18 GMT
• Title: Beyond Accuracy: What Matters in Designing Well-Behaved Models?
• Authors: Robin Hesse, Doğukan Bağcı, Bernt Schiele, Simone Schaub-Meyer, Stefan Roth,
• Abstract summary: We show that vision-language models exhibit high fairness on ImageNet-1k classification and strong robustness against domain changes.<n>We conclude our study by introducing the QUBA score, a novel metric that ranks models across multiple dimensions of quality.
• Score: 53.252827682118955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning has become an essential part of computer vision, with deep neural networks (DNNs) excelling in predictive performance. However, they often fall short in other critical quality dimensions, such as robustness, calibration, or fairness. While existing studies have focused on a subset of these quality dimensions, none have explored a more general form of "well-behavedness" of DNNs. With this work, we address this gap by simultaneously studying nine different quality dimensions for image classification. Through a large-scale study, we provide a bird's-eye view by analyzing 326 backbone models and how different training paradigms and model architectures affect the quality dimensions. We reveal various new insights such that (i) vision-language models exhibit high fairness on ImageNet-1k classification and strong robustness against domain changes; (ii) self-supervised learning is an effective training paradigm to improve almost all considered quality dimensions; and (iii) the training dataset size is a major driver for most of the quality dimensions. We conclude our study by introducing the QUBA score (Quality Understanding Beyond Accuracy), a novel metric that ranks models across multiple dimensions of quality, enabling tailored recommendations based on specific user needs.

Related papers

• Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions [65.89403417819764]
  We quantify the impact of design choices on language model capabilities.<n>By incorporating features besides model size and number of training tokens, we can achieve a relative 3-28% increase in ability to predict downstream performance.
  arXiv  Detail & Related papers  (2025-03-05T19:46:04Z)
• Q-Ground: Image Quality Grounding with Large Multi-modality Models [61.72022069880346]
  We introduce Q-Ground, the first framework aimed at tackling fine-scale visual quality grounding. Q-Ground combines large multi-modality models with detailed visual quality analysis. Central to our contribution is the introduction of the QGround-100K dataset.
  arXiv  Detail & Related papers  (2024-07-24T06:42:46Z)
• Opinion-Unaware Blind Image Quality Assessment using Multi-Scale Deep Feature Statistics [54.08757792080732]
  We propose integrating deep features from pre-trained visual models with a statistical analysis model to achieve opinion-unaware BIQA (OU-BIQA) Our proposed model exhibits superior consistency with human visual perception compared to state-of-the-art BIQA models.
  arXiv  Detail & Related papers  (2024-05-29T06:09:34Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Learning Generalizable Perceptual Representations for Data-Efficient No-Reference Image Quality Assessment [7.291687946822539]
  A major drawback of state-of-the-art NR-IQA techniques is their reliance on a large number of human annotations. We enable the learning of low-level quality features to distortion types by introducing a novel quality-aware contrastive loss. We design zero-shot quality predictions from both pathways in a completely blind setting.
  arXiv  Detail & Related papers  (2023-12-08T05:24:21Z)
• Inter-model Interpretability: Self-supervised Models as a Case Study [0.2578242050187029]
  We build on a recent interpretability technique called Dissect to introduce textitinter-model interpretability We project 13 top-performing self-supervised models into a Learned Concepts Embedding space that reveals proximities among models from the perspective of learned concepts. The experiment allowed us to categorize the models into three categories and revealed for the first time the type of visual concepts different tasks requires.
  arXiv  Detail & Related papers  (2022-07-24T22:50:18Z)
• Self-Supervised Visual Representation Learning Using Lightweight Architectures [0.0]
  In self-supervised learning, a model is trained to solve a pretext task, using a data set whose annotations are created by a machine. We critically examine the most notable pretext tasks to extract features from image data. We study the performance of various self-supervised techniques keeping all other parameters uniform.
  arXiv  Detail & Related papers  (2021-10-21T14:13:10Z)
• Learning to Resize Images for Computer Vision Tasks [15.381549764216134]
  We show that the typical linear resizer can be replaced with learned resizers that can substantially improve performance. Our learned image resizer is jointly trained with a baseline vision model. We show that the proposed resizer can also be useful for fine-tuning the classification baselines for other vision tasks.
  arXiv  Detail & Related papers  (2021-03-17T23:43:44Z)
• Empirical Performance Analysis of Conventional Deep Learning Models for Recognition of Objects in 2-D Images [0.0]
  We have varied parameters like learning rate, filter size, the number of hidden layers, stride size and the activation function among others to analyze the performance of the model. The model classifies images into 3 categories, namely, cars, faces and aeroplanes.
  arXiv  Detail & Related papers  (2020-11-12T20:14:03Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.