Improving Agent Interactions in Virtual Environments with Language Models

• URL: http://arxiv.org/abs/2402.05440v1
• Date: Thu, 8 Feb 2024 06:34:11 GMT
• Title: Improving Agent Interactions in Virtual Environments with Language Models
• Authors: Jack Zhang
• Abstract summary: This research focuses on a collective building assignment in the Minecraft dataset. We employ language modeling to enhance task understanding through state-of-the-art methods.
• Score: 0.9790236766474201
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Enhancing AI systems with efficient communication skills for effective human assistance necessitates proactive initiatives from the system side to discern specific circumstances and interact aptly. This research focuses on a collective building assignment in the Minecraft dataset, employing language modeling to enhance task understanding through state-of-the-art methods. These models focus on grounding multi-modal understanding and task-oriented dialogue comprehension tasks, providing insights into their interpretative and responsive capabilities. Our experimental results showcase a substantial improvement over existing methods, indicating a promising direction for future research in this domain.

Related papers

• Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation [70.52558242336988]
  We focus on predicting engagement in dyadic interactions by scrutinizing verbal and non-verbal cues, aiming to detect signs of disinterest or confusion. In this work, we collect a dataset featuring 34 participants engaged in casual dyadic conversations, each providing self-reported engagement ratings at the end of each conversation. We introduce a novel fusion strategy using Large Language Models (LLMs) to integrate multiple behavior modalities into a multimodal transcript''
  arXiv  Detail & Related papers  (2024-09-13T18:28:12Z)
• Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments [42.06453257292203]
  We propose a hierarchical framework that combines the deep language comprehension of large language models with the adaptive action-execution capabilities of reinforcement learning agents. We have demonstrated the effectiveness of our approach in two different environments: in IGLU, where agents are instructed to build structures, and in Crafter, where agents perform tasks and interact with objects in the surrounding environment according to language commands.
  arXiv  Detail & Related papers  (2024-07-12T14:19:36Z)
• Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models [56.257840490146]
  ConCue is a novel approach for improving visual feature extraction in HOI detection. We develop a transformer-based feature extraction module with a multi-tower architecture that integrates contextual cues into both instance and interaction detectors.
  arXiv  Detail & Related papers  (2023-11-26T09:11:32Z)
• Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements [28.630542719519855]
  This work empirically investigates the performance of large language models (LLMs) in generating empathetic responses. Extensive experiments show that LLMs can significantly benefit from our proposed methods and is able to achieve state-of-the-art performance in both automatic and human evaluations.
  arXiv  Detail & Related papers  (2023-10-08T12:21:24Z)
• Self-Explanation Prompting Improves Dialogue Understanding in Large Language Models [52.24756457516834]
  We propose a novel "Self-Explanation" prompting strategy to enhance the comprehension abilities of Large Language Models (LLMs) This task-agnostic approach requires the model to analyze each dialogue utterance before task execution, thereby improving performance across various dialogue-centric tasks. Experimental results from six benchmark datasets confirm that our method consistently outperforms other zero-shot prompts and matches or exceeds the efficacy of few-shot prompts.
  arXiv  Detail & Related papers  (2023-09-22T15:41:34Z)
• Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling [0.0]
  Our proposed method employs language modeling to enhance task understanding through state-of-the-art (SOTA) methods using language models. Our experimental results provide compelling evidence of the superiority of our proposed method.
  arXiv  Detail & Related papers  (2023-06-21T17:17:09Z)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate [95.10641301155232]
  We present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.
  arXiv  Detail & Related papers  (2023-05-23T17:55:11Z)
• Interactive Natural Language Processing [67.87925315773924]
  Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP. This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
  arXiv  Detail & Related papers  (2023-05-22T17:18:29Z)
• Knowledge-enhanced Agents for Interactive Text Games [16.055119735473017]
  We propose a knowledge-injection framework for improved functional grounding of agents in text-based games. We consider two forms of domain knowledge that we inject into learning-based agents: memory of previous correct actions and affordances of relevant objects in the environment. Our framework supports two representative model classes: reinforcement learning agents and language model agents.
  arXiv  Detail & Related papers  (2023-05-08T23:31:39Z)
• Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues [88.73739515457116]
  We introduce four self-supervised tasks including next session prediction, utterance restoration, incoherence detection and consistency discrimination. We jointly train the PLM-based response selection model with these auxiliary tasks in a multi-task manner. Experiment results indicate that the proposed auxiliary self-supervised tasks bring significant improvement for multi-turn response selection.
  arXiv  Detail & Related papers  (2020-09-14T08:44:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.