Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

• URL: http://arxiv.org/abs/2408.01417v1
• Date: Fri, 2 Aug 2024 17:51:57 GMT
• Title: Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs
• Authors: Yilun Hua, Yoav Artzi,
• Abstract summary: Humans spontaneously use increasingly efficient language as interactions progress, by adapting and forming ad-hoc conventions. It remains unexplored whether multimodal large language models (MLLMs) similarly increase communication efficiency during interactions. We introduce ICCA, an automated framework to evaluate such conversational adaptation as an in-context behavior in MLLMs.
• Score: 14.997971970162743
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans spontaneously use increasingly efficient language as interactions progress, by adapting and forming ad-hoc conventions. This phenomenon has been studied extensively using reference games, showing properties of human language that go beyond relaying intents. It remains unexplored whether multimodal large language models (MLLMs) similarly increase communication efficiency during interactions, and what mechanisms they may adopt for this purpose. We introduce ICCA, an automated framework to evaluate such conversational adaptation as an in-context behavior in MLLMs. We evaluate several state-of-the-art MLLMs, and observe that while they may understand the increasingly efficient language of their interlocutor, they do not spontaneously make their own language more efficient over time. This latter ability can only be elicited in some models (e.g., GPT-4) with heavy-handed prompting. This shows that this property of linguistic interaction does not arise from current training regimes, even though it is a common hallmark of human language. ICCA is available at https://github.com/lil-lab/ICCA.

Related papers

• NatLan: Native Language Prompting Facilitates Knowledge Elicitation Through Language Trigger Provision and Domain Trigger Retention [4.051397588437236]
  We analogize the dominant language of MLLMs to the native language of humans and use two human cognitive features to interpret translate-then-answer methods. We propose Native Language Prompting (NatLan), employing a Multi-MLLM collaboration strategy and introducing an additional role-enhanced domain-specific MLLM.
  arXiv  Detail & Related papers  (2024-08-07T04:49:38Z)
• Language Model Can Listen While Speaking [17.584201137311286]
  Listen-while-speaking language model (LSLM) is an end-to-end system equipped with both listening and speaking channels. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems.
  arXiv  Detail & Related papers  (2024-08-05T16:47:22Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, effectively being crosslingual? This study evaluates six state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue [73.69510478736483]
  Large language models (LLMs) can generate fluent, coherent, and diverse responses. However, they lack a crucial ability: communication skills. This article aims to empower LLMs with communication skills through inner monologues. Experimental results show that the proposed CSIM strategy improves the backbone models and outperforms the baselines.
  arXiv  Detail & Related papers  (2023-11-13T16:19:42Z)
• Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations [70.7884839812069]
  Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue.
  arXiv  Detail & Related papers  (2023-11-09T18:45:16Z)
• Let Models Speak Ciphers: Multiagent Debate through Embeddings [84.20336971784495]
  We introduce CIPHER (Communicative Inter-Model Protocol Through Embedding Representation) to address this issue. By deviating from natural language, CIPHER offers an advantage of encoding a broader spectrum of information without any modification to the model weights. This showcases the superiority and robustness of embeddings as an alternative "language" for communication among LLMs.
  arXiv  Detail & Related papers  (2023-10-10T03:06:38Z)
• Improving Language Plasticity via Pretraining with Active Forgetting [63.36484652568976]
  We propose to use an active forgetting mechanism during pretraining, as a simple way of creating PLMs that can quickly adapt to new languages. Experiments with RoBERTa show that models pretrained with our forgetting mechanism demonstrate faster convergence during language adaptation.
  arXiv  Detail & Related papers  (2023-07-03T17:12:44Z)
• Multi-lingual agents through multi-headed neural networks [0.0]
  This paper focuses on cooperative Multi-Agent Reinforcement Learning. In this context, multiple distinct and incompatible languages can emerge. We take inspiration from the Continual Learning literature and equip our agents with multi-headed neural networks which enable our agents to be multi-lingual.
  arXiv  Detail & Related papers  (2021-11-22T11:39:42Z)
• Self-play for Data Efficient Language Acquisition [20.86261546611472]
  We exploit the symmetric nature of communication in order to improve the efficiency and quality of language acquisition in learning agents. We show that using self-play as a substitute for direct supervision enables the agent to transfer its knowledge across roles.
  arXiv  Detail & Related papers  (2020-10-10T02:09:19Z)
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment [59.995385574274785]
  We show that, contrary to previous belief, negative interference also impacts low-resource languages. We present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference.
  arXiv  Detail & Related papers  (2020-10-06T20:48:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.