Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Study

• URL: http://arxiv.org/abs/2605.15599v1
• Date: Fri, 15 May 2026 04:14:16 GMT
• Title: Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Study
• Abstract summary: We compare four frozen ViT-B/16 encoders trained with supervised classification, contrastive learning (SigLIP2), masked reconstruction (MAE), and self-distillation (DINOv3)<n>Supervised and contrastive encoders provide the strongest linear separability (logistic AUC: 0.768 and 0.735; SVM AUC: 0.739 and 0.697)<n>We find that DINOv3 underperforms across probe families in this domain.
• Score: 35.18016233072556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Extreme low-data fine-grained classification is common in expert domains where labeling is expensive, yet practitioners still need principled guidance for selecting pretrained encoders. We study emerald inclusion grading with a custom dataset of labeled images across three classes and ask: under matched backbone capacity, how does pretraining objective affect downstream representation quality? We compare four frozen ViT-B/16 encoders trained with supervised classification, contrastive learning (SigLIP2), masked reconstruction (MAE), and self-distillation (DINOv3), and evaluate them with leave-one-out cross-validation using linear and nonlinear probes. To control statistical noise in the low-N regime, we use permutation testing (N=1000) on macro one-vs-rest AUC. Supervised and contrastive encoders provide the strongest linear separability (logistic AUC: 0.768 and 0.735; SVM AUC: 0.739 and 0.697), while MAE improves under nonlinear probes (XGBoost AUC: 0.713). We find that DINOv3 underperforms across probe families in this domain. These results support a practical recommendation for extreme low-data FGVC: prioritize margin-enforcing pretraining objectives when data scarcity restricts probing to linear decision rules, and consider reconstruction-style encoders when nonlinear classifiers are feasible given dataset constraints.

Related papers

• When Does Deep Representation Learning Help Single-Cell Clustering? A Sensitivity-Aware Diagnostic Benchmark for Biomedical AI Pipelines [0.0]
  Single-cell ribonucleic acid sequencing (scRNA-seq) is a foundational technology for precision-medicine.<n>Is an additional deep representation stage worth its compute and tuning cost?<n>We address this question with a diagnostic benchmark of nine clustering pipelines on ten real datasets.
  arXiv  Detail & Related papers  (2026-07-28T04:48:47Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Sharpness Aware Surrogate Training for Spiking Neural Networks [0.0]
  Surrogate gradients are a standard tool for training spiking neural networks (SNNs)<n>But conventional hard forward or surrogate backward training couples a nonsmooth forward model with a biased estimator.<n>We apply sharpness aware Surrogate Training (SAST), which applies sharpness aware contractionmization (SAM) to surrogate forward SNN trained by backpropagation.
  arXiv  Detail & Related papers  (2026-03-14T01:26:26Z)
• Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning [17.384089089363382]
  We identify a root cause that existing methods overlook: the uniform penalization of errors.<n>Current approaches treat all incorrect rollouts within a group identically.<n>We propose the Asymmetric Confidence-aware Error Penalty (ACE)
  arXiv  Detail & Related papers  (2026-02-24T22:46:43Z)
• Unsupervised Feature Selection via Robust Autoencoder and Adaptive Graph Learning [4.371255245299254]
  Unsupervised feature selection (UFS) aims to simultaneously cluster data and identify the most discriminative features.<n>We propose the Robust Autoencoder-based Unsupervised Feature Selection (RAEUFS) model, which leverages a deep autoencoder to learn nonlinear feature representations.<n>Our method outperforms state-of-the-art UFS approaches in both clean and outlier-contaminated data settings.
  arXiv  Detail & Related papers  (2025-12-21T12:42:37Z)
• Learning Sparse Label Couplings for Multilabel Chest X-Ray Diagnosis [0.5735035463793009]
  We study multilabel classification of chest X-rays and present a simple, strong pipeline built on SE-ResNeXt101 $(32 times 4d)$.<n>The backbone is finetuned for 14 thoracic findings with a sigmoid head, trained using Multilabel Iterative Stratification (MIS) for robust cross-validation.<n>We propose a lightweight Label-Graph Refinement module placed after the classifier: given per-label probabilities, it learns a sparse, trainable inter-label coupling matrix.
  arXiv  Detail & Related papers  (2025-11-11T03:38:01Z)
• Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values [53.72318444646282]
  We propose Reinforcement Learning with Explicit Human Values (RLEV)<n>RLEV aligns Large Language Model (LLM) optimization directly with quantifiable human value signals.<n>We show RLEV consistently outperforms correctness-only baselines across multiple RL algorithms and model scales.
  arXiv  Detail & Related papers  (2025-10-23T04:15:22Z)
• ECG Latent Feature Extraction with Autoencoders for Downstream Prediction Tasks [2.2616169634370076]
  The electrocardiogram (ECG) is an inexpensive and widely available tool for cardiac assessment.<n>Despite its standardized format and small file size, the high complexity and inter-individual variability of ECG signals make it challenging to use in deep learning models.<n>This study addresses these challenges by exploring feature generation methods from representative beat ECGs.<n>We introduce three novel Variational Autoencoder (VAE) variants-Stochastic Autoencoder (SAE), Annealed beta-VAE (A beta-VAE), and Cyclical beta VAE (C beta-VAE)-and compare their effectiveness in maintaining
  arXiv  Detail & Related papers  (2025-07-31T19:37:05Z)
• A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques [63.10251271444959]
  Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. We conduct an in-depth investigation of the impact of popular choices for three crucial axes. Our setup spanning over 300 experiments reveals consistent trends and unexpected findings.
  arXiv  Detail & Related papers  (2024-06-07T12:25:51Z)
• Class-Imbalanced Semi-Supervised Learning for Large-Scale Point Cloud Semantic Segmentation via Decoupling Optimization [64.36097398869774]
  Semi-supervised learning (SSL) has been an active research topic for large-scale 3D scene understanding. The existing SSL-based methods suffer from severe training bias due to class imbalance and long-tail distributions of the point cloud data. We introduce a new decoupling optimization framework, which disentangles feature representation learning and classifier in an alternative optimization manner to shift the bias decision boundary effectively.
  arXiv  Detail & Related papers  (2024-01-13T04:16:40Z)
• DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection [72.25697820290502]
  This work introduces a straightforward and efficient strategy to identify potential novel classes through zero-shot classification. We refer to this approach as the self-training strategy, which enhances recall and accuracy for novel classes without requiring extra annotations, datasets, and re-training. Empirical evaluations on three datasets, including LVIS, V3Det, and COCO, demonstrate significant improvements over the baseline performance.
  arXiv  Detail & Related papers  (2023-10-02T17:52:24Z)
• Few-Shot Non-Parametric Learning with Deep Latent Variable Model [50.746273235463754]
  We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV) NPC-LV is a learning framework for any dataset with abundant unlabeled data but very few labeled ones. We show that NPC-LV outperforms supervised methods on all three datasets on image classification in low data regime.
  arXiv  Detail & Related papers  (2022-06-23T09:35:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.