Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization

• URL: http://arxiv.org/abs/2604.03417v1
• Date: Fri, 03 Apr 2026 19:30:34 GMT
• Title: Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization
• Abstract summary: We explore the use of large language models (LLMs) and vision models (VMs) as proxies for human judgment.<n>We show that prompt engineering that combines few-shot examples and diverse input formats, such as image embeddings, significantly improves LLM-human alignment.<n>Our results suggest that AI can feasibly serve as a scalable proxy for human labelers.
• Score: 47.53880889741314
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Network visualization has traditionally relied on heuristic metrics, such as stress, under the assumption that optimizing them leads to aesthetic and informative layouts. However, no single metric consistently produces the most effective results. A data-driven alternative is to learn from human preferences, where annotators select their favored visualization among multiple layouts of the same graphs. These human-preference labels can then be used to train a generative model that approximates human aesthetic preferences. However, obtaining human labels at scale is costly and time-consuming. As a result, this generative approach has so far been tested only with machine-labeled data. In this paper, we explore the use of large language models (LLMs) and vision models (VMs) as proxies for human judgment. Through a carefully designed user study involving 27 participants, we curated a large set of human preference labels. We used this data both to better understand human preferences and to bootstrap LLM/VM labelers. We show that prompt engineering that combines few-shot examples and diverse input formats, such as image embeddings, significantly improves LLM-human alignment, and additional filtering by the confidence score of the LLM pushes the alignment to human-human levels. Furthermore, we demonstrate that carefully trained VMs can achieve VM-human alignment at a level comparable to that between human annotators. Our results suggest that AI can feasibly serve as a scalable proxy for human labelers.

Related papers

• AI Outperforms Humans in Personalized Image Aesthetics Assessment via LLM-Based Interviews and Semantic Feature Extraction [2.905751301655124]
  Accurately predicting individual aesthetic evaluation for images is a fundamental challenge for AI.<n>We develop an integrated DL-LLM system that elicits aesthetic preferences through semi-structured interviews.<n>Our results show that the proposed system outperforms all of them, with particularly strong performance on highly-rated images.
  arXiv  Detail & Related papers  (2026-05-14T12:24:23Z)
• HumanLLM: Towards Personalized Understanding and Simulation of Human Nature [72.55730315685837]
  HumanLLM is a foundation model designed for personalized understanding and simulation of individuals.<n>We first construct the Cognitive Genome, a large-scale corpus curated from real-world user data on platforms like Reddit, Twitter, Blogger, and Amazon.<n>We then formulate diverse learning tasks and perform supervised fine-tuning to empower the model to predict a wide range of individualized human behaviors, thoughts, and experiences.
  arXiv  Detail & Related papers  (2026-01-22T09:27:27Z)
• Improving Large Vision and Language Models by Learning from a Panel of Peers [27.83658413272528]
  We propose a novel Panel-of-Peers learning framework inspired by collaborative learning among humans.<n>By simulating a peer review system, our models generate, assess, and refine outputs in response to a curated set of prompts.<n>Our experiments show significant improvement across multiple benchmarks, demonstrating the potential of peer evaluations as a scalable alternative to self-supervised alignment.
  arXiv  Detail & Related papers  (2025-09-01T16:43:48Z)
• A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learning [67.72413262980272]
  Pre-trained vision models (PVMs) are fundamental to modern robotics, yet their optimal configuration remains unclear.<n>We develop SlotMIM, a method that induces object-centric representations by introducing a semantic bottleneck.<n>Our approach achieves significant improvements over prior work in image recognition, scene understanding, and robot learning evaluations.
  arXiv  Detail & Related papers  (2025-03-10T06:18:31Z)
• Online Learning from Strategic Human Feedback in LLM Fine-Tuning [16.35495567193046]
  Reinforcement learning from human feedback (RLHF) has become an essential step in fine-tuning large language models (LLMs) to align them with human preferences.<n>We study online learning mechanisms against strategic human labelers in the LLM fine-tuning process.<n>We formulate a new dynamic Bayesian game and dynamically adjust human labelers' weights in the preference aggregation.
  arXiv  Detail & Related papers  (2024-12-22T02:43:07Z)
• Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment [73.14105098897696]
  We propose Representation-Aligned Preference-based Learning (RAPL) to learn visual rewards from significantly less human preference feedback.<n>RAPL focuses on fine-tuning pre-trained vision encoders to align with the end-user's visual representation and then constructs a dense visual reward via feature matching.<n>We show that RAPL can learn rewards aligned with human preferences, more efficiently uses preference data, and generalizes across robot embodiments.
  arXiv  Detail & Related papers  (2024-12-06T08:04:02Z)
• Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback [87.37721254914476]
  We introduce HyPER, a Hybrid Preference routER that defers an annotation to either humans or language models (LMs)<n>We show that the selected hybrid mixture of synthetic and direct human preferences using HyPER achieves better RM performance compared to using either one exclusively by 7-13% on RewardBench.<n>We also analyze features from HyPER and find that prompts with moderate safety concerns or complexity benefit the most from human feedback.
  arXiv  Detail & Related papers  (2024-10-24T20:04:15Z)
• Dissecting Human and LLM Preferences [80.55271307662365]
  We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. We show that preference-based evaluation can be intentionally manipulated.
  arXiv  Detail & Related papers  (2024-02-17T14:34:31Z)
• VisAlign: Dataset for Measuring the Degree of Alignment between AI and Humans in Visual Perception [32.376529738717736]
  We propose a new dataset for measuring AI-human visual alignment in terms of image classification. Our dataset consists of three groups of samples, namely Must-Act (i.e., Must-Classify), Must-Abstain, and Uncertain. We analyze the visual alignment and reliability of five popular visual perception models and seven abstention methods.
  arXiv  Detail & Related papers  (2023-08-03T04:04:03Z)
• Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets [90.61266099147053]
  We investigate efficient annotation strategies for collecting multi-class classification labels for a large collection of images. We propose modifications and best practices aimed at minimizing human labeling effort. Simulated experiments on a 125k image subset of the ImageNet100 show that it can be annotated to 80% top-1 accuracy with 0.35 annotations per image on average.
  arXiv  Detail & Related papers  (2021-04-26T16:29:32Z)
• A Machine Learning Approach for Predicting Human Preference for Graph Layouts [46.31441679172231]
  We present the first machine learning approach for predicting human preference for graph layouts. Experimental results using the ground truth human preference data sets show that our model can successfully predict human preference for graph layouts.
  arXiv  Detail & Related papers  (2021-03-01T23:54:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.