Context Sensitivity Improves Human-Machine Visual Alignment

• URL: http://arxiv.org/abs/2604.13883v1
• Date: Wed, 15 Apr 2026 13:47:08 GMT
• Title: Context Sensitivity Improves Human-Machine Visual Alignment
• Abstract summary: We propose a method for context-sensitive similarity computation from neural network embeddings.<n>We achieve up to a 15% improvement in odd-one-out accuracy over a context-insensitive model.
• Score: 21.48392018729352
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern machine learning models typically represent inputs as fixed points in a high-dimensional embedding space. While this approach has been proven powerful for a wide range of downstream tasks, it fundamentally differs from the way humans process information. Because humans are constantly adapting to their environment, they represent objects and their relationships in a highly context-sensitive manner. To address this gap, we propose a method for context-sensitive similarity computation from neural network embeddings, applied to modeling a triplet odd-one-out task with an anchor image serving as simultaneous context. Modeling context enables us to achieve up to a 15% improvement in odd-one-out accuracy over a context-insensitive model. We find that this improvement is consistent across both original and "human-aligned" vision foundation models.

Related papers

• AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis [50.793806818677716]
  AffordGrasp produces physically stable and semantically faithful human grasps with high precision.<n>AffordGrasp integrates an affordance-aware latent representation of hand poses with a dual-conditioning diffusion process.<n>We evaluate AffordGrasp across four instruction-augmented benchmarks derived from HO-3D, OakInk, GRAB, and AffordPose.
  arXiv  Detail & Related papers  (2026-03-09T06:56:35Z)
• A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Data [0.0]
  We introduce a computational framework designed to model core aspects of human referential interpretation.<n>We evaluate the model on the Stanford Repeated Reference Game corpus.<n>Results suggest that relatively simple perceptual-linguistic alignment mechanisms can yield human-competitive behavior.
  arXiv  Detail & Related papers  (2026-02-23T07:20:11Z)
• Multimodal Representation Alignment for Cross-modal Information Retrieval [12.42313654539524]
  Different machine learning models can represent the same underlying concept in different ways.<n>This variability is particularly valuable for in-the-wild multimodal retrieval, where the objective is to identify the corresponding representation in one modality given another as input.<n>In this work, we first investigate the geometric relationships between visual and textual embeddings derived from both vision-language models and combined unimodal models.<n>We then align these representations using four standard similarity metrics as well as two learned ones, implemented via neural networks.
  arXiv  Detail & Related papers  (2025-06-10T13:16:26Z)
• Exploiting Contextual Uncertainty of Visual Data for Efficient Training of Deep Models [0.65268245109828]
  We introduce the notion of contextual diversity for active learning CDAL. We propose a data repair algorithm to curate contextually fair data to reduce model bias. We are working on developing image retrieval system for wildlife camera trap images and reliable warning system for poor quality rural roads.
  arXiv  Detail & Related papers  (2024-11-04T09:43:33Z)
• Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models [65.82564074712836]
  We introduce DIFfusionHOI, a new HOI detector shedding light on text-to-image diffusion models. We first devise an inversion-based strategy to learn the expression of relation patterns between humans and objects in embedding space. These learned relation embeddings then serve as textual prompts, to steer diffusion models generate images that depict specific interactions.
  arXiv  Detail & Related papers  (2024-10-26T12:00:33Z)
• Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks [62.48782506095565]
  We show that due to the greedy nature of learning in deep neural networks, models tend to rely on just one modality while under-fitting the other modalities. We propose an algorithm to balance the conditional learning speeds between modalities during training and demonstrate that it indeed addresses the issue of greedy learning.
  arXiv  Detail & Related papers  (2022-02-10T20:11:21Z)
• Image Synthesis via Semantic Composition [74.68191130898805]
  We present a novel approach to synthesize realistic images based on their semantic layouts. It hypothesizes that for objects with similar appearance, they share similar representation. Our method establishes dependencies between regions according to their appearance correlation, yielding both spatially variant and associated representations.
  arXiv  Detail & Related papers  (2021-09-15T02:26:07Z)
• Scene Synthesis via Uncertainty-Driven Attribute Synchronization [52.31834816911887]
  This paper introduces a novel neural scene synthesis approach that can capture diverse feature patterns of 3D scenes. Our method combines the strength of both neural network-based and conventional scene synthesis approaches.
  arXiv  Detail & Related papers  (2021-08-30T19:45:07Z)
• Robust and Interpretable Grounding of Spatial References with Relation Networks [40.42540299023808]
  Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. We develop effective models for understanding spatial references in text that are robust and interpretable.
  arXiv  Detail & Related papers  (2020-05-02T04:11:33Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.