Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions

• URL: http://arxiv.org/abs/2408.08780v3
• Date: Thu, 22 Aug 2024 02:52:28 GMT
• Title: Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions
• Authors: Chenming Tang, Zhixiang Wang, Yunfang Wu,
• Abstract summary: We propose an ensemble prompt framework to describe the selection criteria of multiple in-context examples. Preliminary experiments on machine translation (MT) across six translation directions confirm that this framework boosts ICL perfromance.
• Score: 13.87098305304058
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the help of in-context learning (ICL), large language models (LLMs) have achieved impressive performance across various tasks. However, the function of descriptive instructions during ICL remains under-explored. In this work, we propose an ensemble prompt framework to describe the selection criteria of multiple in-context examples, and preliminary experiments on machine translation (MT) across six translation directions confirm that this framework boosts ICL perfromance. But to our surprise, LLMs might not necessarily care what the descriptions actually say, and the performance gain is primarily caused by the ensemble format, since the framework could lead to improvement even with random descriptive nouns. We further apply this new ensemble prompt on a range of commonsense, math, logical reasoning and hallucination tasks with three LLMs and achieve promising results, suggesting again that designing a proper prompt format would be much more effective and efficient than paying effort into specific descriptions. Our code will be publicly available once this paper is published.

Related papers

• PromptExp: Multi-granularity Prompt Explanation of Large Language Models [16.259208045898415]
  We introduce PromptExp, a framework for multi-granularity prompt explanations by aggregating token-level insights. PromptExp supports both white-box and black-box explanations and extends explanations to higher granularity levels. We evaluate PromptExp in case studies such as sentiment analysis, showing the perturbation-based approach performs best.
  arXiv  Detail & Related papers  (2024-10-16T22:25:15Z)
• Large Language Models are Pattern Matchers: Editing Semi-Structured and Structured Documents with ChatGPT [0.0]
  This paper investigates if Large Language Models (LLMs) can be applied for editing structured and semi-structured documents with minimal effort. ChatGPT demonstrates a strong ability to recognize and process the structure of annotated documents.
  arXiv  Detail & Related papers  (2024-09-12T03:41:39Z)
• INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning [59.07490387145391]
  Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. We introduce a novel instruction tuning dataset, INTERS, encompassing 20 tasks across three fundamental IR categories.
  arXiv  Detail & Related papers  (2024-01-12T12:10:28Z)
• AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations [52.43593893122206]
  Alignedcot is an in-context learning technique for invoking Large Language Models. It achieves consistent and correct step-wise prompts in zero-shot scenarios. We conduct experiments on mathematical reasoning and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-11-22T17:24:21Z)
• Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning [50.00090601424348]
  Large language models (LLMs) have shown remarkable capabilities in various natural language understanding tasks. We propose EASE, an Explanation-Aware Soft Ensemble framework to empower in-context learning with LLMs.
  arXiv  Detail & Related papers  (2023-11-13T06:13:38Z)
• The language of prompting: What linguistic properties make a prompt successful? [13.034603322224548]
  LLMs can be prompted to achieve impressive zero-shot or few-shot performance in many NLP tasks. Yet, we still lack a systematic understanding of how linguistic properties of prompts correlate with task performance. We investigate both grammatical properties such as mood, tense, aspect and modality, as well as lexico-semantic variation through the use of synonyms.
  arXiv  Detail & Related papers  (2023-11-03T15:03:36Z)
• Can Large Language Models Understand Real-World Complex Instructions? [54.86632921036983]
  Large language models (LLMs) can understand human instructions, but struggle with complex instructions. Existing benchmarks are insufficient to assess LLMs' ability to understand complex instructions. We propose CELLO, a benchmark for evaluating LLMs' ability to follow complex instructions systematically.
  arXiv  Detail & Related papers  (2023-09-17T04:18:39Z)
• Waffling around for Performance: Visual Classification with Random Words and Broad Concepts [121.60918966567657]
  WaffleCLIP is a framework for zero-shot visual classification which simply replaces LLM-generated descriptors with random character and word descriptors. We conduct an extensive experimental study on the impact and shortcomings of additional semantics introduced with LLM-generated descriptors.
  arXiv  Detail & Related papers  (2023-06-12T17:59:48Z)
• ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction [56.790794611002106]
  Large language models (LLMs) have demonstrated remarkable results in various natural language processing (NLP) tasks with in-context learning. We propose a simple but effective in-context learning framework called ICL-D3IE. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations.
  arXiv  Detail & Related papers  (2023-03-09T06:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.