Evaluating 5W3H Structured Prompting for Intent Alignment in Human-AI Interaction

• URL: http://arxiv.org/abs/2603.18976v1
• Date: Thu, 19 Mar 2026 14:41:06 GMT
• Title: Evaluating 5W3H Structured Prompting for Intent Alignment in Human-AI Interaction
• Abstract summary: We evaluate PPS, a framework for structured intent representation in human-AI interaction.<n>We study 60 tasks in three domains (business, technical, and travel)<n>We find structured intent representations can improve alignment and usability in human-AI interaction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Natural language prompts often suffer from intent transmission loss: the gap between what users actually need and what they communicate to AI systems. We evaluate PPS (Prompt Protocol Specification), a 5W3H-based framework for structured intent representation in human-AI interaction. In a controlled three-condition study across 60 tasks in three domains (business, technical, and travel), three large language models (DeepSeek-V3, Qwen-Max, and Kimi), and three prompt conditions - (A) simple prompts, (B) raw PPS JSON, and (C) natural-language-rendered PPS - we collect 540 AI-generated outputs evaluated by an LLM judge. We introduce goal_alignment, a user-intent-centered evaluation dimension, and find that rendered PPS outperforms both simple prompts and raw JSON on this metric. PPS gains are task-dependent: gains are large in high-ambiguity business analysis tasks but reverse in low-ambiguity travel planning. We also identify a measurement asymmetry in standard LLM evaluation, where unconstrained prompts can inflate constraint adherence scores and mask the practical value of structured prompting. A preliminary retrospective survey (N = 20) further suggests a 66.1% reduction in follow-up prompts required, from 3.33 to 1.13 rounds. These findings suggest that structured intent representations can improve alignment and usability in human-AI interaction, especially in tasks where user intent is inherently ambiguous.

Related papers

• Intent Signal Theory: A Computational Framework for Intent-State Control in Human-AI Interaction [0.585480332059272]
  Current AI interaction models treat the prompt as the primary object of exchange, omitting a critical layer: the user's latent source intent.<n>Here we introduce Intent Signal Theory (IST), a computational framework that formalises this missing intent layer.<n>IST distinguishes four objects routinely conflated: latent source intent (I*), observable intent proxy (I-hat), encoded carrier (P), and model output (O)
  arXiv  Detail & Related papers  (2026-05-24T13:10:33Z)
• Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? [74.69723255239663]
  Grounded Personality Reasoning requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding.<n> MM-OCEAN is produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs.<n>The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%.
  arXiv  Detail & Related papers  (2026-05-21T07:42:47Z)
• PromptNCE: Pointwise Mutual Information Predictions Using Only LLMs and Contrastive Estimation Prompts [3.015315464409211]
  Estimating mutual information from text usually requires training a task-specific critic, which limits its use in low-data settings.<n>We ask whether large language models can instead estimate pointwise mutual information zero-shot, using only prompts and elicited probabilities.<n>We introduce a benchmark with human-derived ground-truth PMI across three publicly available datasets, and evaluate five information-theoretic prompting-based estimators.
  arXiv  Detail & Related papers  (2026-05-20T22:10:54Z)
• Structured Intent as a Protocol-Like Communication Layer: Cross-Model Robustness, Framework Comparison, and the Weak-Model Compensation Effect [0.0]
  This paper investigates how reliably structured intent representations preserve user goals across different AI models, languages, and prompting frameworks.<n>We find that structured prompting substantially reduces cross-language score variance relative to unstructured baselines.<n>In a user study, AI-expanded 5W3H prompts reduce interaction rounds by 60 percent and increase user satisfaction from 3.16 to 4.04.
  arXiv  Detail & Related papers  (2026-03-31T16:20:28Z)
• Does Structured Intent Representation Generalize? A Cross-Language, Cross-Model Empirical Study of 5W3H Prompting [0.0]
  We study PPS, a 5W3H-based framework for structured intent representation in human-AI interaction.<n>We find that AI-expanded 5W3H prompts show no statistically significant difference in goal alignment from manually crafted 5W3H prompts.<n>We also show that unstructured prompts exhibit a systematic dual-inflation bias.
  arXiv  Detail & Related papers  (2026-03-26T12:29:42Z)
• Retrieval-Augmented Guardrails for AI-Drafted Patient-Portal Messages: Error Taxonomy Construction and Large-Scale Evaluation [5.555479009357263]
  Asynchronous patient-clinician messaging via EHR portals is a growing source of clinician workload.<n>Our contributions are threefold: (1) we introduce a clinically grounded error ontology comprising 5 domains and 59 granular error codes; (2) we develop a retrieval-augmented evaluation pipeline; and (3) we provide a two-stage prompting architecture using DSPy to enable scalable, interpretable, and hierarchical error detection.
  arXiv  Detail & Related papers  (2025-09-26T16:42:43Z)
• Evaluating LLM Alignment on Personality Inference from Real-World Interview Data [7.061237517845673]
  Large Language Models (LLMs) are increasingly deployed in roles requiring nuanced psychological understanding.<n>Their ability to interpret human personality traits, a critical aspect of such applications, remains unexplored.<n>We introduce a novel benchmark comprising semi-structured interview transcripts paired with validated continuous Big Five trait scores.
  arXiv  Detail & Related papers  (2025-09-16T16:54:35Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Quantifying and Optimizing Global Faithfulness in Persona-driven Role-playing [37.92922713921964]
  Persona-driven role-playing (PRP) aims to build AI characters that can respond to user queries by faithfully sticking with all persona statements. This paper presents a pioneering exploration to quantify PRP faithfulness as a fine-grained and explainable criterion, which also serves as a reliable reference for optimization.
  arXiv  Detail & Related papers  (2024-05-13T13:21:35Z)
• Query Performance Prediction using Relevance Judgments Generated by Large Language Models [53.97064615557883]
  We propose a new Query performance prediction (QPP) framework using automatically generated relevance judgments (QPP-GenRE)<n>QPP-GenRE decomposes QPP into independent subtasks of predicting relevance of each item in a ranked list to a given query.<n>We predict an item's relevance by using open-source large language models (LLMs) to ensure scientific relevance.
  arXiv  Detail & Related papers  (2024-04-01T09:33:05Z)
• Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs [58.620269228776294]
  We propose a task-agnostic framework for resolving ambiguity by asking users clarifying questions. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. We find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs.
  arXiv  Detail & Related papers  (2023-11-16T00:18:50Z)
• SOUL: Towards Sentiment and Opinion Understanding of Language [96.74878032417054]
  We propose a new task called Sentiment and Opinion Understanding of Language (SOUL) SOUL aims to evaluate sentiment understanding through two subtasks: Review (RC) and Justification Generation (JG)
  arXiv  Detail & Related papers  (2023-10-27T06:48:48Z)
• Evaluating Human-Language Model Interaction [79.33022878034627]
  We develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems. We design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. We find that better non-interactive performance does not always translate to better human-LM interaction.
  arXiv  Detail & Related papers  (2022-12-19T18:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.