Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot

• URL: http://arxiv.org/abs/2605.23819v1
• Date: Fri, 22 May 2026 16:21:25 GMT
• Title: Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spot
• Abstract summary: A central question in computational vision is whether human-like visual representations are better explained by discriminative or generative learning.<n>We address this confound using Joint Energy-Based Models (JEMs), which interpolate continuously between discriminative and generative training within a fixed architecture.
• Score: 18.610144165897264
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A central question in computational vision is whether human-like visual representations are better explained by discriminative or generative learning. Existing comparisons, however, often confound the learning objective with architecture, scale, and training data, leaving open whether the objective itself drives alignment. We address this confound using Joint Energy-Based Models (JEMs), which interpolate continuously between discriminative and generative training within a fixed architecture. By varying a single mixing coefficient, we isolate the effect of the learning objective and evaluate the resulting models across six human-alignment benchmarks spanning perceptual similarity, gloss perception, human response uncertainty, robustness, shape-texture cue conflict, and diagnostic feature attribution. Across this diverse suite, human alignment is consistently maximized at intermediate points of the generative-discriminative continuum, rather than at either endpoint. Hybrid JEMs combine the categorical structure induced by discriminative learning with the sensitivity to input structure induced by generative learning, yielding more human-like behavior across multiple levels of vision. These results suggest that the generative-discriminative dichotomy is the wrong axis for understanding human-aligned vision: alignment emerges not from choosing one objective over the other, but from balancing both.

Related papers

• Do Machines Fail Like Humans? A Human-Centred Out-of-Distribution Spectrum for Mapping Error Alignment [1.974820485832244]
  We propose a human-centred framework that redefines the degree of OOD as a spectrum of human perceptual difficulty.<n>We apply this framework to object recognition and reveal unique, regime-dependent model-human alignment rankings and profiles across deep learning architectures.
  arXiv  Detail & Related papers  (2026-03-08T04:51:39Z)
• Affinity Contrastive Learning for Skeleton-based Human Activity Understanding [36.788675803693486]
  ACLNet is an Affinity Contrastive Learning Network that explores the intricate clustering relationships among human activity classes.<n>We propose an affinity metric to refine similarity measurements, thereby forming activity superclasses that provide more informative contrastive signals.<n>In addition, we employ a margin-based contrastive strategy to improve the separation of hard positive and negative samples within classes.
  arXiv  Detail & Related papers  (2026-01-23T12:20:36Z)
• Understanding and evaluating computer vision models through the lens of counterfactuals [2.2819712364325047]
  This thesis develops frameworks that use counterfactuals to explain, audit, and mitigate bias in vision classifiers and generative models.<n>By systematically altering semantically meaningful attributes while holding others fixed, these methods uncover spurious correlations.<n>These contributions show counterfactuals as a unifying lens for interpretability, fairness, and causality in both discriminative and generative models.
  arXiv  Detail & Related papers  (2025-08-28T15:11:49Z)
• Unveiling the Human-like Similarities of Automatic Facial Expression Recognition: An Empirical Exploration through Explainable AI [0.562479170374811]
  This study aims to explore the similarity between deep neural networks and human perception by comparing twelve different networks. We employ an innovative global explainable AI method to generate heatmaps, revealing crucial facial regions for the twelve networks trained on six facial expressions. Findings suggest limited alignment between human and AI facial expression recognition, with network architectures influencing the similarity.
  arXiv  Detail & Related papers  (2024-01-22T10:52:02Z)
• Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? [90.35044668396591]
  A recurring theme in machine learning is algorithmic monoculture: the same systems, or systems that share components, are deployed by multiple decision-makers. We propose the component-sharing hypothesis: if decision-makers share components like training data or specific models, then they will produce more homogeneous outcomes. We test this hypothesis on algorithmic fairness benchmarks, demonstrating that sharing training data reliably exacerbates homogenization. We conclude with philosophical analyses of and societal challenges for outcome homogenization, with an eye towards implications for deployed machine learning systems.
  arXiv  Detail & Related papers  (2022-11-25T09:33:11Z)
• Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
  Existing contrastive learning methods suffer from very low learning efficiency. Under-clustering and over-clustering problems are major obstacles to learning efficiency. We propose a novel self-supervised learning framework using a median triplet loss.
  arXiv  Detail & Related papers  (2021-04-18T07:47:10Z)
• Deep Clustering by Semantic Contrastive Learning [67.28140787010447]
  We introduce a novel variant called Semantic Contrastive Learning (SCL) It explores the characteristics of both conventional contrastive learning and deep clustering. It can amplify the strengths of contrastive learning and deep clustering in a unified approach.
  arXiv  Detail & Related papers  (2021-03-03T20:20:48Z)
• Contextual Heterogeneous Graph Network for Human-Object Interaction Detection [63.37410475907447]
  This work proposes a heterogeneous graph network that models humans and objects as different kinds of nodes. In addition, a graph attention mechanism based on the intra-class context and inter-class context is exploited to improve the learning.
  arXiv  Detail & Related papers  (2020-10-20T04:20:33Z)
• Fully Unsupervised Person Re-identification viaSelective Contrastive Learning [58.5284246878277]
  Person re-identification (ReID) aims at searching the same identity person among images captured by various cameras. We propose a novel selective contrastive learning framework for unsupervised feature learning. Experimental results demonstrate the superiority of our method in unsupervised person ReID compared with the state-of-the-arts.
  arXiv  Detail & Related papers  (2020-10-15T09:09:23Z)
• DRG: Dual Relation Graph for Human-Object Interaction Detection [65.50707710054141]
  We tackle the challenging problem of human-object interaction (HOI) detection. Existing methods either recognize the interaction of each human-object pair in isolation or perform joint inference based on complex appearance-based features. In this paper, we leverage an abstract spatial-semantic representation to describe each human-object pair and aggregate the contextual information of the scene via a dual relation graph.
  arXiv  Detail & Related papers  (2020-08-26T17:59:40Z)
• Seeing eye-to-eye? A comparison of object recognition performance in humans and deep convolutional neural networks under image manipulation [0.0]
  This study aims towards a behavioral comparison of visual core object recognition performance between humans and feedforward neural networks. Analyses of accuracy revealed that humans not only outperform DCNNs on all conditions, but also display significantly greater robustness towards shape and most notably color alterations.
  arXiv  Detail & Related papers  (2020-07-13T10:26:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.