What if Othello-Playing Language Models Could See?

• URL: http://arxiv.org/abs/2507.14520v1
• Date: Sat, 19 Jul 2025 07:47:55 GMT
• Title: What if Othello-Playing Language Models Could See?
• Authors: Xinyi Chen, Yifei Yuan, Jiaang Li, Serge Belongie, Maarten de Rijke, Anders Søgaard,
• Abstract summary: We introduce VISOTHELLO, a multi-modal model trained on move histories and board images.<n>We find that multi-modal training improves both performance and the robustness of internal representations.
• Score: 81.14765602011055
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Language models are often said to face a symbol grounding problem. While some argue that world understanding can emerge from text alone, others suggest grounded learning is more efficient. We explore this through Othello, where the board state defines a simplified, rule-based world. Building on prior work, we introduce VISOTHELLO, a multi-modal model trained on move histories and board images. Using next-move prediction, we compare it to mono-modal baselines and test robustness to semantically irrelevant perturbations. We find that multi-modal training improves both performance and the robustness of internal representations. These results suggest that grounding language in visual input helps models infer structured world representations.

Related papers

• Visual Grounding Helps Learn Word Meanings in Low-Data Regimes [47.7950860342515]
  Modern neural language models (LMs) are powerful tools for modeling human sentence production and comprehension. But to achieve these results, LMs must be trained in distinctly un-human-like ways. Do models trained more naturalistically -- with grounded supervision -- exhibit more humanlike language learning? We investigate this question in the context of word learning, a key sub-task in language acquisition.
  arXiv  Detail & Related papers  (2023-10-20T03:33:36Z)
• Learning to Model the World with Language [100.76069091703505]
  To interact with humans and act in the world, agents need to understand the range of language that people use and relate it to the visual world. Our key idea is that agents should interpret such diverse language as a signal that helps them predict the future. We instantiate this in Dynalang, an agent that learns a multimodal world model to predict future text and image representations.
  arXiv  Detail & Related papers  (2023-07-31T17:57:49Z)
• Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models [57.08925810659545]
  We conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models. Our empirical observations suggest that vision-and-language models are better at label prediction tasks. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
  arXiv  Detail & Related papers  (2022-12-01T05:00:18Z)
• CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings [2.8478710949588284]
  CoLLIE is a model for continual learning of how language is grounded in vision. It learns a transformation function that adjusts the language embeddings when needed to accommodate new language use. We show that CoLLIE can efficiently learn and generalize from only a few examples.
  arXiv  Detail & Related papers  (2021-11-15T18:54:58Z)
• Does Vision-and-Language Pretraining Improve Lexical Grounding? [25.357191933430627]
  Vision-and-Language models are trained jointly on text and image or video data. It is not yet known how the internal linguistic representations themselves compare to their text-only counterparts.
  arXiv  Detail & Related papers  (2021-09-21T15:12:39Z)
• Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition [80.446770909975]
  Linguistic knowledge is of great benefit to scene text recognition. How to effectively model linguistic rules in end-to-end deep networks remains a research challenge. We propose an autonomous, bidirectional and iterative ABINet for scene text recognition.
  arXiv  Detail & Related papers  (2021-03-11T06:47:45Z)
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision [110.66085917826648]
  We develop a technique that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images. "vokenization" is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks.
  arXiv  Detail & Related papers  (2020-10-14T02:11:51Z)
• Visual Grounding in Video for Unsupervised Word Translation [91.47607488740647]
  We use visual grounding to improve unsupervised word mapping between languages. We learn embeddings from unpaired instructional videos narrated in the native language. We apply these methods to translate words from English to French, Korean, and Japanese.
  arXiv  Detail & Related papers  (2020-03-11T02:03:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.