Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines

• URL: http://arxiv.org/abs/2504.07840v1
• Date: Thu, 10 Apr 2025 15:20:43 GMT
• Title: Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines
• Authors: Cansu Koyuturk, Emily Theophilou, Sabrina Patania, Gregor Donabauer, Andrea Martinenghi, Chiara Antico, Alessia Telari, Alessia Testa, Sathya Bursic, Franca Garzotto, Davinia Hernandez-Leo, Udo Kruschwitz, Davide Taibi, Simona Amenta, Martin Ruskov, Dimitri Ognibene,
• Abstract summary: This study investigates learner-AI interactions through an educational experiment in which participants receive structured guidance on effective prompting.<n>To assess user behavior and prompting efficacy, we analyze a dataset of 642 interactions from 107 users.<n>Our findings provide a deeper understanding of how users engage with Large Language Models and the role of structured prompting guidance in enhancing AI-assisted communication.
• Score: 9.834055425277874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have transformed human-computer interaction by enabling natural language-based communication with AI-powered chatbots. These models are designed to be intuitive and user-friendly, allowing users to articulate requests with minimal effort. However, despite their accessibility, studies reveal that users often struggle with effective prompting, resulting in inefficient responses. Existing research has highlighted both the limitations of LLMs in interpreting vague or poorly structured prompts and the difficulties users face in crafting precise queries. This study investigates learner-AI interactions through an educational experiment in which participants receive structured guidance on effective prompting. We introduce and compare three types of prompting guidelines: a task-specific framework developed through a structured methodology and two baseline approaches. To assess user behavior and prompting efficacy, we analyze a dataset of 642 interactions from 107 users. Using Von NeuMidas, an extended pragmatic annotation schema for LLM interaction analysis, we categorize common prompting errors and identify recurring behavioral patterns. We then evaluate the impact of different guidelines by examining changes in user behavior, adherence to prompting strategies, and the overall quality of AI-generated responses. Our findings provide a deeper understanding of how users engage with LLMs and the role of structured prompting guidance in enhancing AI-assisted communication. By comparing different instructional frameworks, we offer insights into more effective approaches for improving user competency in AI interactions, with implications for AI literacy, chatbot usability, and the design of more responsive AI systems.

Related papers

• Conversational AI as a Coding Assistant: Understanding Programmers' Interactions with and Expectations from Large Language Models for Coding [5.064404027153094]
  Conversational AI interfaces powered by large language models (LLMs) are increasingly used as coding assistants.<n>This study investigates programmers' usage patterns, perceptions, and interaction strategies when engaging with LLM-driven coding assistants.
  arXiv  Detail & Related papers  (2025-03-14T15:06:07Z)
• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.<n>Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes.<n>We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs [49.18567856499736]
  We investigate whether large language models (LLMs) can be supportive of open-ended dialogue tutoring.<n>We apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue.<n>We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• Enabling Real-Time Conversations with Minimal Training Costs [61.80370154101649]
  This paper presents a new duplex decoding approach that enhances large language models with duplex ability, requiring minimal training. Experimental results indicate that our proposed method significantly enhances the naturalness and human-likeness of user-AI interactions with minimal training costs.
  arXiv  Detail & Related papers  (2024-09-18T06:27:26Z)
• Constraining Participation: Affordances of Feedback Features in Interfaces to Large Language Models [49.74265453289855]
  Large language models (LLMs) are now accessible to anyone with a computer, a web browser, and an internet connection via browser-based interfaces. This paper examines the affordances of interactive feedback features in ChatGPT's interface, analysing how they shape user input and participation in iteration.
  arXiv  Detail & Related papers  (2024-08-27T13:50:37Z)
• Modulating Language Model Experiences through Frictions [56.17593192325438]
  Over-consumption of language model outputs risks propagating unchecked errors in the short-term and damaging human capabilities for critical thinking in the long-term. We propose selective frictions for language model experiences, inspired by behavioral science interventions, to dampen misuse.
  arXiv  Detail & Related papers  (2024-06-24T16:31:11Z)
• Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations [17.409790984399052]
  This paper explores the impact of multi-modal interactions, which incorporate images and audio alongside text, on user engagement. Our findings reveal a significant enhancement in user engagement with multi-modal interactions compared to text-only dialogues. Results suggest that multi-modal interactions optimize cognitive processing and facilitate richer information comprehension.
  arXiv  Detail & Related papers  (2024-06-21T09:26:55Z)
• Rethinking ChatGPT's Success: Usability and Cognitive Behaviors Enabled by Auto-regressive LLMs' Prompting [5.344199202349884]
  We analyze the structure of modalities within both two types of Large Language Models and six task-specific channels during deployment. We examine the stimulation of diverse cognitive behaviors in LLMs through the adoption of free-form text and verbal contexts.
  arXiv  Detail & Related papers  (2024-05-17T00:19:41Z)
• Re-Reading Improves Reasoning in Large Language Models [87.46256176508376]
  We introduce a simple, yet general and effective prompting method, Re2, to enhance the reasoning capabilities of off-the-shelf Large Language Models (LLMs) Unlike most thought-eliciting prompting methods, such as Chain-of-Thought (CoT), Re2 shifts the focus to the input by processing questions twice, thereby enhancing the understanding process. We evaluate Re2 on extensive reasoning benchmarks across 14 datasets, spanning 112 experiments, to validate its effectiveness and generality.
  arXiv  Detail & Related papers  (2023-09-12T14:36:23Z)
• Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection [57.13665112065285]
  Human-Object Interaction (HOI) detection is a challenging computer vision task. We present a framework that enhances HOI detection by incorporating structured text knowledge.
  arXiv  Detail & Related papers  (2023-07-25T14:20:52Z)
• Interactive Natural Language Processing [67.87925315773924]
  Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP. This paper offers a comprehensive survey of iNLP, starting by proposing a unified definition and framework of the concept.
  arXiv  Detail & Related papers  (2023-05-22T17:18:29Z)
• Semantic Interactive Learning for Text Classification: A Constructive Approach for Contextual Interactions [0.0]
  We propose a novel interaction framework called Semantic Interactive Learning for the text domain. We frame the problem of incorporating constructive and contextual feedback into the learner as a task to find an architecture that enables more semantic alignment between humans and machines. We introduce a technique called SemanticPush that is effective for translating conceptual corrections of humans to non-extrapolating training examples.
  arXiv  Detail & Related papers  (2022-09-07T08:13:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.