What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?

• URL: http://arxiv.org/abs/2607.16214v1
• Date: Fri, 22 May 2026 16:43:14 GMT
• Title: What Makes Linguistic Representations Good Models of High-Level Visual Perception in the Human Brain?
• Abstract summary: We studied how images are described and which language models are used to embed them.<n>We found that both the content of image captions and the LM used to represent them influence brain- and behaviour-modelling performance.
• Score: 27.868453876654286
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image descriptions represented with language models (LMs) predict human brain responses to naturalistic images in high-level visual regions, but the factors driving this predictivity remain unclear. To investigate this, we systematically studied how images are described and which language models are used to embed those descriptions. For a common set of images, we considered six caption types -- including human-annotated and multiple machine-generated captions -- differing along several dimensions. Each caption was represented with five LMs, spanning autoregressive LMs trained to predict upcoming words and text embedders, i.e., LMs fine-tuned on semantic tasks requiring sentence/document-level representations. Machine-generated captions yielded significant brain predictivity and alignment, often surpassing human-annotated captions used in previous work. Across caption types, text embedders consistently outperformed autoregressive LMs, a pattern replicated when measuring behavioural alignment with image-similarity judgments. Analyses of caption representations from different model layers further revealed that both brain predictivity and behavioural alignment peak at intermediate network depth, shortly after a point thought to mark the emergence of syntactic and semantic structure. Altogether, our results demonstrate that both the content of image captions and the LM used to represent them influence brain- and behaviour-modelling performance, establishing caption embeddings as a useful tool for studying high-level visual perception.

Related papers

• Visual Instruction-Finetuned Language Model for Versatile Brain MR Image Tasks [1.4770902450080214]
  LLaBIT (Large Language Model for Brain Image Translation) extends the visual reasoning of LLMs to clinically meaningful tasks in the brain MRI domain.<n>We evaluate our method on five brain MRI datasets across four distinct tasks.<n>Our model not only demonstrated superior performance across all tasks but also outperformed specialized, task-specific models in direct comparisons.
  arXiv  Detail & Related papers  (2026-04-03T05:39:47Z)
• The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping [1.5767445615203355]
  Visual Iconicity Challenge adapts psycholinguistic measures to evaluate vision-language models.<n>We assess 13 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands.<n>Models with stronger phonological form prediction correlate better with human iconicity judgment.
  arXiv  Detail & Related papers  (2025-10-09T17:21:59Z)
• Modelling Multimodal Integration in Human Concept Processing with Vision-Language Models [7.511284868070148]
  We investigate whether integration of visuo-linguistic information leads to representations that are more aligned with human brain activity.<n>Our findings indicate an advantage of multimodal models in predicting human brain activations.
  arXiv  Detail & Related papers  (2024-07-25T10:08:37Z)
• Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning [64.1316997189396]
  We present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets.
  arXiv  Detail & Related papers  (2024-03-21T17:58:56Z)
• Visual Grounding Helps Learn Word Meanings in Low-Data Regimes [47.7950860342515]
  Modern neural language models (LMs) are powerful tools for modeling human sentence production and comprehension. But to achieve these results, LMs must be trained in distinctly un-human-like ways. Do models trained more naturalistically -- with grounded supervision -- exhibit more humanlike language learning? We investigate this question in the context of word learning, a key sub-task in language acquisition.
  arXiv  Detail & Related papers  (2023-10-20T03:33:36Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)
• DeViL: Decoding Vision features into Language [53.88202366696955]
  Post-hoc explanation methods have often been criticised for abstracting away the decision-making process of deep neural networks. In this work, we would like to provide natural language descriptions for what different layers of a vision backbone have learned. We train a transformer network to translate individual image features of any vision layer into a prompt that a separate off-the-shelf language model decodes into natural language.
  arXiv  Detail & Related papers  (2023-09-04T13:59:55Z)
• Universal Multimodal Representation for Language Understanding [110.98786673598015]
  This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively.
  arXiv  Detail & Related papers  (2023-01-09T13:54:11Z)
• Linearly Mapping from Image to Text Space [22.290431852705662]
  We show that conceptual representations learned by text-only models are functionally equivalent to those learned by models trained on vision tasks. We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining. We find that all three encoders perform equally well at transferring visual property information to the language model.
  arXiv  Detail & Related papers  (2022-09-30T01:17:18Z)
• Visual representations in the human brain are aligned with large language models [7.779248296336383]
  We show that large language models (LLMs) are beneficial for modelling the complex visual information extracted by the brain from natural scenes. We then train deep neural network models to transform image inputs into LLM representations.
  arXiv  Detail & Related papers  (2022-09-23T17:34:33Z)
• Evaluating language-biased image classification based on semantic representations [13.508894957080777]
  Humans show language-biased image recognition for a word-embedded image, known as picture-word interference. Similar to humans, recent artificial models jointly trained on texts and images, e.g., OpenAI CLIP, show language-biased image classification.
  arXiv  Detail & Related papers  (2022-01-26T15:46:36Z)
• Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning [50.08729005865331]
  This paper develops a plug-and-play hierarchical-topic-guided image paragraph generation framework. To capture the correlations between the image and text at multiple levels of abstraction, we design a variational inference network. To guide the paragraph generation, the learned hierarchical topics and visual features are integrated into the language model.
  arXiv  Detail & Related papers  (2021-05-10T06:55:39Z)
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language [148.0843278195794]
  We propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions.
  arXiv  Detail & Related papers  (2020-11-18T20:21:19Z)
• Improving Image Captioning with Better Use of Captions [65.39641077768488]
  We present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences.
  arXiv  Detail & Related papers  (2020-06-21T14:10:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.