From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning

• URL: http://arxiv.org/abs/2505.17117v3
• Date: Mon, 30 Jun 2025 21:22:39 GMT
• Title: From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
• Authors: Chen Shani, Dan Jurafsky, Yann LeCun, Ravid Shwartz-Ziv,
• Abstract summary: Humans organize knowledge into compact categories through semantic compression.<n>Large Language Models (LLMs) demonstrate remarkable linguistic abilities.<n>But whether their internal representations strike a human-like trade-off between compression and semantic fidelity is unclear.
• Score: 52.32745233116143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans organize knowledge into compact categories through semantic compression by mapping diverse instances to abstract representations while preserving meaning (e.g., robin and blue jay are both birds; most birds can fly). These concepts reflect a trade-off between expressive fidelity and representational simplicity. Large Language Models (LLMs) demonstrate remarkable linguistic abilities, yet whether their internal representations strike a human-like trade-off between compression and semantic fidelity is unclear. We introduce a novel information-theoretic framework, drawing from Rate-Distortion Theory and the Information Bottleneck principle, to quantitatively compare these strategies. Analyzing token embeddings from a diverse suite of LLMs against seminal human categorization benchmarks, we uncover key divergences. While LLMs form broad conceptual categories that align with human judgment, they struggle to capture the fine-grained semantic distinctions crucial for human understanding. More fundamentally, LLMs demonstrate a strong bias towards aggressive statistical compression, whereas human conceptual systems appear to prioritize adaptive nuance and contextual richness, even if this results in lower compressional efficiency by our measures. These findings illuminate critical differences between current AI and human cognitive architectures, guiding pathways toward LLMs with more human-aligned conceptual representations.

Related papers

• Emerging Human-like Strategies for Semantic Memory Foraging in Large Language Models [0.8749675983608171]
  Both humans and Large Language Models (LLMs) store a vast repository of semantic memories.<n>In humans, efficient and strategic access to this memory store is a critical foundation for a variety of cognitive functions.
  arXiv  Detail & Related papers  (2026-03-02T12:55:51Z)
• UniCog: Uncovering Cognitive Abilities of LLMs through Latent Mind Space Analysis [69.50752734049985]
  A growing body of research suggests that the cognitive processes of large language models (LLMs) differ fundamentally from those of humans.<n>We propose UniCog, a unified framework that analyzes LLM cognition via a latent mind space.
  arXiv  Detail & Related papers  (2026-01-25T16:19:00Z)
• A Brain-like Synergistic Core in LLMs Drives Behaviour and Learning [50.68188138112555]
  We show that large language models spontaneously develop synergistic cores.<n>We find that areas in middle layers exhibit synergistic processing while early and late layers rely on redundancy.<n>This convergence suggests that synergistic information processing is a fundamental property of intelligence.
  arXiv  Detail & Related papers  (2026-01-11T10:48:35Z)
• From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilers [14.983442449498739]
  This study investigates whether and how Large Language Models can model the correlational structure of human psychological traits from minimal quantitative inputs.<n>We prompted various LLMs with Big Five Personality Scale responses from 816 human individuals to role-play their responses on nine other psychological scales.<n>LLMs demonstrated remarkable accuracy in capturing human psychological structure.
  arXiv  Detail & Related papers  (2025-11-05T06:51:13Z)
• Uncovering the Computational Ingredients of Human-Like Representations in LLMs [8.00888290370075]
  It remains unclear which of these ingredients are most crucial for building models that develop human-like representations.<n>Most current benchmarks are not suited to measuring representational alignment between humans and models.
  arXiv  Detail & Related papers  (2025-10-01T15:37:19Z)
• LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition [14.683883775425821]
  This paper proposes a novel method for understanding human intents from multimodal signals.<n>The method harnesses the expansive knowledge of large language models (LLMs) to establish semantic foundations.<n>Experiments on multimodal intent and dialogue act tasks demonstrate LGSRR's superiority over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-09-01T10:18:47Z)
• Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes [54.93980123979578]
  We introduce Latent Preference Coding (LPC), a novel framework that models the implicit factors as well as their combinations behind holistic preferences.<n>LPC seamlessly integrates with various offline alignment algorithms, automatically inferring the underlying factors and their importance from data.
  arXiv  Detail & Related papers  (2025-05-08T06:59:06Z)
• Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition [89.50068130832635]
  Self-Improving cognition (SIcog) is a self-learning framework for constructing next-generation foundation MLLMs by multimodal knowledge.<n>We propose Chain-of-Description for step-by-step visual understanding and integrate structured Chain-of-Thought (CoT) reasoning to support in-depth multimodal reasoning.<n>Experiments demonstrate SIcog's effectiveness in developing MLLMs with enhanced multimodal cognition.
  arXiv  Detail & Related papers  (2025-03-16T00:25:13Z)
• How Deep is Love in LLMs' Hearts? Exploring Semantic Size in Human-like Cognition [75.11808682808065]
  This study investigates whether large language models (LLMs) exhibit similar tendencies in understanding semantic size.<n>Our findings reveal that multi-modal training is crucial for LLMs to achieve more human-like understanding.<n> Lastly, we examine whether LLMs are influenced by attention-grabbing headlines with larger semantic sizes in a real-world web shopping scenario.
  arXiv  Detail & Related papers  (2025-03-01T03:35:56Z)
• Human-like conceptual representations emerge from language prediction [72.5875173689788]
  Large language models (LLMs) trained exclusively through next-token prediction over language data exhibit remarkably human-like behaviors.<n>Are these models developing concepts akin to humans, and if so, how are such concepts represented and organized?<n>Our results demonstrate that LLMs can flexibly derive concepts from linguistic descriptions in relation to contextual cues about other concepts.<n>These findings establish that structured, human-like conceptual representations can naturally emerge from language prediction without real-world grounding.
  arXiv  Detail & Related papers  (2025-01-21T23:54:17Z)
• VladVA: Discriminative Fine-tuning of LVLMs [67.14293827774827]
  Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the de facto approach for discriminative vision-language representation learning.<n>We propose to combine "the best of both worlds": a new training approach for discriminative fine-tuning of LVLMs.
  arXiv  Detail & Related papers  (2024-12-05T17:54:27Z)
• Human-like object concept representations emerge naturally in multimodal large language models [24.003766123531545]
  We combined behavioral and neuroimaging analyses to explore the relationship between object concept representations in Large Language Models (LLMs) and human cognition.<n>Our findings advance the understanding of machine intelligence and inform the development of more human-like artificial cognitive systems.
  arXiv  Detail & Related papers  (2024-07-01T08:17:19Z)
• LLMs as Models for Analogical Reasoning [14.412456982731467]
  Analogical reasoning is fundamental to human cognition and learning.<n>Recent studies have shown that large language models can sometimes match humans in analogical reasoning tasks.
  arXiv  Detail & Related papers  (2024-06-19T20:07:37Z)
• Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Brain [12.92793034617015]
  We show that as large language models (LLMs) achieve higher performance on benchmark tasks, they become more brain-like. We also show the importance of contextual information in improving model performance and brain similarity.
  arXiv  Detail & Related papers  (2024-01-31T08:48:35Z)
• Interpreting Pretrained Language Models via Concept Bottlenecks [55.47515772358389]
  Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. The lack of interpretability due to their black-box'' nature poses challenges for responsible implementation. We propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans.
  arXiv  Detail & Related papers  (2023-11-08T20:41:18Z)
• Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learning [22.201878275784246]
  Focus in Explainable AI is shifting from explanations defined in terms of low-level elements, such as input features, to explanations encoded in terms of interpretable concepts learned from data. How to reliably acquire such concepts is, however, still fundamentally unclear. We propose a mathematical framework for acquiring interpretable representations suitable for both post-hoc explainers and concept-based neural networks.
  arXiv  Detail & Related papers  (2023-09-14T14:26:20Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)
• The Fluidity of Concept Representations in Human Brain Signals [0.0]
  We analyze the discriminability of concrete and abstract concepts in fMRI data. We argue that fluid concept representations lead to more realistic models of human language processing.
  arXiv  Detail & Related papers  (2020-02-20T17:31:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.