Controllable Navigation Instruction Generation with Chain of Thought Prompting

• URL: http://arxiv.org/abs/2407.07433v2
• Date: Tue, 16 Jul 2024 10:09:34 GMT
• Title: Controllable Navigation Instruction Generation with Chain of Thought Prompting
• Authors: Xianghao Kong, Jinyu Chen, Wenguan Wang, Hang Su, Xiaolin Hu, Yi Yang, Si Liu,
• Abstract summary: We propose C-Instructor, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation. C-Instructor renders generated instructions more accessible to follow and offers greater controllability over the manipulation of landmark objects.
• Score: 74.34604350917273
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Instruction generation is a vital and multidisciplinary research area with broad applications. Existing instruction generation models are limited to generating instructions in a single style from a particular dataset, and the style and content of generated instructions cannot be controlled. Moreover, most existing instruction generation methods also disregard the spatial modeling of the navigation environment. Leveraging the capabilities of Large Language Models (LLMs), we propose C-Instructor, which utilizes the chain-of-thought-style prompt for style-controllable and content-controllable instruction generation. Firstly, we propose a Chain of Thought with Landmarks (CoTL) mechanism, which guides the LLM to identify key landmarks and then generate complete instructions. CoTL renders generated instructions more accessible to follow and offers greater controllability over the manipulation of landmark objects. Furthermore, we present a Spatial Topology Modeling Task to facilitate the understanding of the spatial structure of the environment. Finally, we introduce a Style-Mixed Training policy, harnessing the prior knowledge of LLMs to enable style control for instruction generation based on different prompts within a single model instance. Extensive experiments demonstrate that instructions generated by C-Instructor outperform those generated by previous methods in text metrics, navigation guidance evaluation, and user studies.

Related papers

• Improving Instruction-Following in Language Models through Activation Steering [58.876600545898675]
  We derive instruction-specific vector representations from language models and use them to steer models accordingly. We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion. Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation.
  arXiv  Detail & Related papers  (2024-10-15T08:38:20Z)
• From Symbolic Tasks to Code Generation: Diversification Yields Better Task Performers [1.6958018695660049]
  We show that a more diverse instruction set, extending beyond code-related tasks, improves the performance of code generation. Our observations suggest that a more diverse semantic space for instruction-tuning sets greatly improves the model's ability to follow instructions and perform tasks.
  arXiv  Detail & Related papers  (2024-05-30T07:54:07Z)
• Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models [153.14575887549088]
  We introduce Generalized Instruction Tuning (called GLAN), a general and scalable method for instruction tuning of Large Language Models (LLMs) GLAN exclusively utilizes a pre-curated taxonomy of human knowledge and capabilities as input and generates large-scale synthetic instruction data across all disciplines. With the fine-grained key concepts detailed in every class session of the syllabus, we are able to generate diverse instructions with a broad coverage across the entire spectrum of human knowledge and skills.
  arXiv  Detail & Related papers  (2024-02-20T15:00:35Z)
• EasyInstruct: An Easy-to-use Instruction Processing Framework for Large Language Models [37.80143756214926]
  EasyInstruct is an easy-to-use instruction processing framework for Large Language Models (LLMs) EasyInstruct modularizes instruction generation, selection, and prompting, while also considering their combination and interaction.
  arXiv  Detail & Related papers  (2024-02-05T14:33:56Z)
• Benchmarking Large Language Models on Controllable Generation under Diversified Instructions [34.89012022437519]
  Large language models (LLMs) have exhibited impressive instruction-following capabilities. It is still unclear whether and to what extent they can respond to explicit constraints that might be entailed in various instructions. We propose a new benchmark CoDI-Eval to evaluate LLMs' responses to instructions with various constraints.
  arXiv  Detail & Related papers  (2024-01-01T07:35:31Z)
• From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning [63.63840740526497]
  We investigate how instruction tuning adjusts pre-trained models with a focus on intrinsic changes. The impact of instruction tuning is then studied by comparing the explanations derived from the pre-trained and instruction-tuned models. Our findings reveal three significant impacts of instruction tuning.
  arXiv  Detail & Related papers  (2023-09-30T21:16:05Z)
• Guiding Large Language Models via Directional Stimulus Prompting [114.84930073977672]
  We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model to generate an auxiliary directional stimulus prompt for each input instance.
  arXiv  Detail & Related papers  (2023-02-22T17:44:15Z)
• Learning Multi-Objective Curricula for Deep Reinforcement Learning [55.27879754113767]
  Various automatic curriculum learning (ACL) methods have been proposed to improve the sample efficiency and final performance of deep reinforcement learning (DRL) In this paper, we propose a unified automatic curriculum learning framework to create multi-objective but coherent curricula. In addition to existing hand-designed curricula paradigms, we further design a flexible memory mechanism to learn an abstract curriculum.
  arXiv  Detail & Related papers  (2021-10-06T19:30:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.