Models Can and Should Embrace the Communicative Nature of Human-Generated Math

• URL: http://arxiv.org/abs/2409.17005v2
• Date: Thu, 31 Oct 2024 17:21:13 GMT
• Title: Models Can and Should Embrace the Communicative Nature of Human-Generated Math
• Authors: Sasha Boguraev, Ben Lipkin, Leonie Weissweiler, Kyle Mahowald,
• Abstract summary: We argue that math data that models are trained on reflects not just idealized mathematical entities but rich communicative intentions. We advocate for AI systems that learn from and represent the communicative intentions latent in human-generated math.
• Score: 13.491107542643839
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Math is constructed by people for people: just as natural language corpora reflect not just propositions but the communicative goals of language users, the math data that models are trained on reflects not just idealized mathematical entities but rich communicative intentions. While there are important advantages to treating math in a purely symbolic manner, we here hypothesize that there are benefits to treating math as situated linguistic communication and that language models are well suited for this goal, in ways that are not fully appreciated. We illustrate these points with two case studies. First, we ran an experiment in which we found that language models interpret the equals sign in a humanlike way -- generating systematically different word problems for the same underlying equation arranged in different ways. Second, we found that language models prefer proofs to be ordered in naturalistic ways, even though other orders would be logically equivalent. We advocate for AI systems that learn from and represent the communicative intentions latent in human-generated math.

Related papers

• On the Thinking-Language Modeling Gap in Large Language Models [68.83670974539108]
  We show that there is a significant gap between the modeling of languages and thoughts.<n>We propose a new prompt technique termed Language-of-Thoughts (LoT) to demonstrate and alleviate this gap.
  arXiv  Detail & Related papers  (2025-05-19T09:31:52Z)
• Non-literal Understanding of Number Words by Language Models [33.24263583093367]
  Humans naturally interpret numbers non-literally, combining context, world knowledge, and speaker intent. We investigate whether large language models (LLMs) interpret numbers similarly, focusing on hyperbole and pragmatic halo effects.
  arXiv  Detail & Related papers  (2025-02-10T07:03:00Z)
• A Distributional Perspective on Word Learning in Neural Language Models [57.41607944290822]
  There are no widely agreed-upon metrics for word learning in language models. We argue that distributional signatures studied in prior work fail to capture key distributional information. We obtain learning trajectories for a selection of small language models we train from scratch.
  arXiv  Detail & Related papers  (2025-02-09T13:15:59Z)
• Perceptions of Linguistic Uncertainty by Language Models and Humans [26.69714008538173]
  We investigate how language models map linguistic expressions of uncertainty to numerical responses. We find that 7 out of 10 models are able to map uncertainty expressions to probabilistic responses in a human-like manner. This sensitivity indicates that language models are substantially more susceptible to bias based on their prior knowledge.
  arXiv  Detail & Related papers  (2024-07-22T17:26:12Z)
• A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models [39.77445889769015]
  We show that, within the PaLM2 family of language models, larger models are more logical than smaller ones. Even the largest models make systematic errors, some of which mirror human reasoning biases. Overall, we find that language models often mimic the human biases included in their training data, but are able to overcome them in some cases.
  arXiv  Detail & Related papers  (2023-11-01T11:13:06Z)
• The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputs [50.32802502923367]
  We study the process of language driving and influencing social reasoning in a probabilistic goal inference domain. We propose a neuro-symbolic model that carries out goal inference from linguistic inputs of agent scenarios. Our model closely matches human response patterns and better predicts human judgements than using an LLM alone.
  arXiv  Detail & Related papers  (2023-06-25T19:38:01Z)
• From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought [124.40905824051079]
  We propose rational meaning construction, a computational framework for language-informed thinking. We frame linguistic meaning as a context-sensitive mapping from natural language into a probabilistic language of thought. We show that LLMs can generate context-sensitive translations that capture pragmatically-appropriate linguistic meanings. We extend our framework to integrate cognitively-motivated symbolic modules.
  arXiv  Detail & Related papers  (2023-06-22T05:14:00Z)
• Evaluating Language Models for Mathematics through Interactions [116.67206980096513]
  We introduce CheckMate, a prototype platform for humans to interact with and evaluate large language models (LLMs) We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics. We derive a taxonomy of human behaviours and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness.
  arXiv  Detail & Related papers  (2023-06-02T17:12:25Z)
• Tree-Based Representation and Generation of Natural and Mathematical Language [77.34726150561087]
  Mathematical language in scientific communications and educational scenarios is important yet relatively understudied. Recent works on mathematical language focus either on representing stand-alone mathematical expressions, or mathematical reasoning in pre-trained natural language models. We propose a series of modifications to existing language models to jointly represent and generate text and math.
  arXiv  Detail & Related papers  (2023-02-15T22:38:34Z)
• Language Models as Inductive Reasoners [125.99461874008703]
  We propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts. We create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language. We provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts.
  arXiv  Detail & Related papers  (2022-12-21T11:12:14Z)
• A fine-grained comparison of pragmatic language understanding in humans and language models [2.231167375820083]
  We compare language models and humans on seven pragmatic phenomena. We find that the largest models achieve high accuracy and match human error patterns. Preliminary evidence that models and humans are sensitive to similar linguistic cues.
  arXiv  Detail & Related papers  (2022-12-13T18:34:59Z)
• Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks [39.39138995087475]
  We ask how much of human-like thinking can be captured by learning statistical patterns in language alone. Our benchmark contains two problem-solving domains (planning and explanation generation) and is designed to require generalization. We find that humans are far more robust than LLMs on this benchmark.
  arXiv  Detail & Related papers  (2022-05-11T18:14:33Z)
• Provable Limitations of Acquiring Meaning from Ungrounded Form: What will Future Language Models Understand? [87.20342701232869]
  We investigate the abilities of ungrounded systems to acquire meaning. We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence. We find that assertions enable semantic emulation if all expressions in the language are referentially transparent. However, if the language uses non-transparent patterns like variable binding, we show that emulation can become an uncomputable problem.
  arXiv  Detail & Related papers  (2021-04-22T01:00:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.