Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation

• URL: http://arxiv.org/abs/2605.19776v2
• Date: Wed, 20 May 2026 07:14:07 GMT
• Title: Preferences Order, Ratings Anchor: From Fused Expert Aesthetic Ground Truth to Self-Distillation
• Abstract summary: We introduce PPaint, a matched dual-protocol benchmark in which 15 domain experts, 5 per category, annotate 150 Chinese paintings.<n>We collect 45,900 pairwise expert judgments through a locally dense preference design alongside the matched ratings.<n>Fusing both signals via two independent preference-to-score methods yields a fused expert ground truth.
• Score: 24.67838359287715
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Pairwise preferences and pointwise ratings are the two dominant annotation protocols in image aesthetic assessment (IAA), yet existing benchmarks adopt only one, leaving their complementarity unmeasured under controlled conditions. We introduce PPaint, a matched dual-protocol benchmark in which 15 domain experts, 5 per category, annotate 150 Chinese paintings under both protocols across five aesthetic dimensions, collecting 45,900 pairwise expert judgments through a locally dense preference design alongside the matched ratings. The matched design reveals complementary strengths: preferences yield more consistent ordinal rankings, while ratings anchor the absolute score scale. Fusing both signals via two independent preference-to-score methods yields a fused expert ground truth on which the two constructions converge to nearly identical scores. The same preference-to-score principle extends to label-free VLM training. PSDistill converts VLM pairwise judgments into calibrated pseudo-scores via an Elo reference pool, and trains the same VLM with confidence-weighted ranking optimization to produce a single-pass aesthetic scorer. Trained on a single painting category, the distilled Qwen3-VL-8B improves mean SRCC from 0.504 to 0.709 across all three categories, outperforming all open-source baselines including the dedicated aesthetic model ArtiMuse and matching closed-source Gemini-3.1-Pro within 0.04 SRCC at single-pass inference cost, with cross-domain transfer further validated on APDDv2. We will release the full PPaint dataset and training code.

Related papers

• Automated Perceptually-Motivated Assessment of Photographic Consistency in Paired Clinical Photographs: Pipeline Development and Internal Evaluation [0.0]
  We developed a pipeline that analyzes pre/post pairs across thirteen calibrated sub-metrics.<n>The pipeline was calibrated on 134 matched within-patient published pre/post pairs and evaluated against identical-image pairs, synthetic-perturbation pairs, and 134 mismatched cross-publication pairs.
  arXiv  Detail & Related papers  (2026-09-14T07:19:17Z)
• Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation [0.639901418243611]
  In real-time colonoscopy, ground-truth annotations are unavailable at inference, so polyp segmentation models can fail silently.<n>We propose Referee-Based Quality Estimation (RBQE), a reference-free framework measuring agreement between a primary segmentation model and an independently trained referee on the same image.
  arXiv  Detail & Related papers  (2026-09-09T17:32:43Z)
• Solution for UCF UrbanTwin LUMPI Track: Sim-to-Real Urban LiDAR 3D Object Detection [54.03436243843709]
  We present our solution to the LUMPI track of the UCF UrbanTwin Sim2Real LiDAR Challenge at the 6th DriveX Workshop, ECCV 2026.<n>The detector must be trained only on synthetic data and is evaluated on 50 held-out real LiDAR frames.<n>A separate 50-frame synthetic submission is evaluated for point-cloud realism.
  arXiv  Detail & Related papers  (2026-09-07T15:03:36Z)
• Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts [7.00753296402747]
  We show that Contrastive Routing Mechanism (CoRM) scores each expert by the gap between its affinity for the incoming token and its affinity for this shared reference state.<n>Our experiments show that CoRM improves average zero-shot accuracy by +0.67 to +1.69 points (Top-1) and +1.38 to +1.77 points (Top-2) over standard Top-k MoE baselines.
  arXiv  Detail & Related papers  (2026-09-01T11:43:41Z)
• PRIME: Mitigating Subgroup Optimization Competition in Shared CTR Top Networks with Plug-in Residual Input-Conditioned Mixture of Expert [16.073701450851008]
  We study the competition on Avazu with 4 models and 4 semantic fields.<n>We introduce PRIME (Plug-in Residual Input-conditioned Mixture of Experts), a Dense-anchored mixture of low-rank residual experts.
  arXiv  Detail & Related papers  (2026-08-31T08:37:44Z)
• Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist [69.81453995715673]
  Existing faithfulness benchmarks rely on simple atomic instructions.<n>We introduce Arena-T2I Hard, a 310-prompt stress benchmark drawn from real arena T2I logs.<n>We propose a dependency-aware checklist reward that decomposes each prompt into a DAG of yes/no questions and zeroes descendants of failed parents.
  arXiv  Detail & Related papers  (2026-06-30T14:17:05Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design [43.31865418601155]
  We release TASTE (Typography, Aesthetics, Spatial, Tone, Etc.): ten professional designers ranked outputs from four current text-to-image models on nine criteria.<n> TASTE places designer agreement on graphic design between food and movie preferences and photo-style image quality.<n>No pre-trained system in our benchmark, including six open-weight VLM judges from 3B to 33B parameters.
  arXiv  Detail & Related papers  (2026-05-20T05:27:05Z)
• CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging [95.02210956333374]
  We propose a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation.<n>On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy.
  arXiv  Detail & Related papers  (2026-05-19T10:59:19Z)
• Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty? [59.923111838399144]
  We introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter.<n>VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task.<n>We find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts.
  arXiv  Detail & Related papers  (2026-05-12T19:33:28Z)
• Open-Set Vein Biometric Recognition with Deep Metric Learning [0.0]
  We rigorously evaluate the computational boundaries of Deep Metric Learning (DML) under strict open-set constraints.<n>Our approach learns discnative L2-normalised embeddings and employs prototype-based matching with a similarity threshold.<n>On the large-scale MMCBNU 6000 benchmark, our best model (ResNet50-CBAM) achieves an OSCR of 0.9945, AUROC of 0.9974, and EER of 1.57%, maintaining high identification accuracy (99.6% Rank-1) while robustly rejecting unknown subjects.
  arXiv  Detail & Related papers  (2026-04-16T11:03:11Z)
• Pseudo Label NCF for Sparse OHC Recommendation: Dual Representation Learning and the Separability Accuracy Trade off [3.558452956820138]
  We study recommendation under extreme interaction sparsity in a survey driven setting.<n>We extend Collaborative Filtering architectures, including Matrix Factorization, Multi Layer Perceptron, and NeuMF, with an auxiliary pseudo label objective derived from survey group feature alignment.<n>The resulting Pseudo Label NCF learns dual embedding spaces: main embeddings for ranking and pseudo label embeddings for semantic alignment.
  arXiv  Detail & Related papers  (2026-03-25T19:21:28Z)
• Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation [3.23600523782706]
  We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning.<n>We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which penalizes disordered reasoning chains.<n>CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.
  arXiv  Detail & Related papers  (2026-03-13T15:48:15Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization [63.169050703903515]
  We propose Aes-R1, a comprehensive aesthetic reasoning framework with reinforcement learning (RL)<n>Aes-R1 integrates a pipeline, AesCoT, to construct and filter high-quality chain-of-thought aesthetic reasoning data.<n>Experiments demonstrate that Aes-R1 improves the backbone's average PLCC/SRCC by 47.9%/34.8%.
  arXiv  Detail & Related papers  (2025-09-26T04:55:00Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.