Drawing Your Programs: Exploring the Applications of Visual-Prompting with GenAI for Teaching and Assessment

• URL: http://arxiv.org/abs/2602.10529v1
• Date: Wed, 11 Feb 2026 04:59:31 GMT
• Title: Drawing Your Programs: Exploring the Applications of Visual-Prompting with GenAI for Teaching and Assessment
• Authors: David H. Smith, S. Moonwara A. Monisha, Annapurna Vadaparty, Leo Porter, Daniel Zingaro,
• Abstract summary: We argue that this text-centric focus overlooks other forms of prompting GenAI models, such as problem decomposition diagrams.<n>We demonstrate that current models are very successful in their ability to generate code from student-constructed diagrams.
• Score: 0.32622301272834514
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When designing a program, both novice programmers and seasoned developers alike often sketch out -- or, perhaps more famously, whiteboard -- their ideas. Yet despite the introduction of natively multimodal Generative AI models, work on Human-GenAI collaborative coding has remained overwhelmingly focused on textual prompts -- largely ignoring the visual and spatial representations that programmers naturally use to reason about and communicate their designs. In this proposal and position paper, we argue and provide tentative evidence that this text-centric focus overlooks other forms of prompting GenAI models, such as problem decomposition diagrams functioning as prompts for code generation in their own right enabling new types of programming activities and assessments. To support this position, we present findings from a large introductory Python programming course, where students constructed decomposition diagrams that were used to prompt GPT-4.1 for code generation. We demonstrate that current models are very successful in their ability to generate code from student-constructed diagrams. We conclude by exploring the implications of embracing multimodal prompting for computing education, particularly in the context of assessment.

Related papers

• JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligence [48.39202336809688]
  We introduce a complete synthesis toolkit to efficiently produce a large-scale, high-quality corpus spanning from standard charts to complex interactive web UIs and code-driven animations.<n>This powers the training of our models, JanusCoder and JanusCoderV, which establish a visual-programmatic interface for generating code from textual instructions, visual inputs, or a combination of both.<n>Our 7B to 14B scale models approaching or even exceeding the performance of commercial models.
  arXiv  Detail & Related papers  (2025-10-27T17:13:49Z)
• Teaching Programming in the Age of Generative AI: Insights from Literature, Pedagogical Proposals, and Student Perspectives [0.0]
  This article aims to review the most relevant studies on how programming content should be taught, learned, and assessed.<n>It proposes enriching teaching and learning methodologies by focusing on code comprehension and execution.<n>It advocates for the use of visual representations of code and visual simulations of its execution as effective tools for teaching, learning, and assessing programming.
  arXiv  Detail & Related papers  (2025-06-30T17:38:27Z)
• Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models [21.267699046566698]
  Students increasingly rely on generative AI tools for programming assistance, often without formal instruction or guidance.<n>This highlights a need to teach students how to effectively interact with AI models.<n>We developed a novel platform for prompt programming that enables authentic dialogue-based interactions.
  arXiv  Detail & Related papers  (2025-03-06T09:56:07Z)
• An Exploratory Study of ML Sketches and Visual Code Assistants [9.451971404904722]
  We build a first prototype of a Visual Code Assistant to get user feedback regarding in-IDE sketch-to-code tools.<n>We analyze developers' mental models by analyzing patterns commonly observed in their sketches when developing an ML workflow.<n>We use an LLM-as-judge setup to score the quality of the generated code, finding that even brief sketching can effectively generate useful code outlines.
  arXiv  Detail & Related papers  (2024-12-17T23:44:45Z)
• Visual Program Distillation with Template-Based Augmentation [58.394564822036195]
  We propose a low-cost visual program distillation method that requires no human-generated program annotations.<n>With a relatively small amount of question/answer data, small language models can generate high-quality specialized visual programs with the added benefit of much faster inference.
  arXiv  Detail & Related papers  (2024-12-11T17:32:21Z)
• Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping [55.98643055756135]
  We introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. We analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs. A user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception.
  arXiv  Detail & Related papers  (2024-10-21T17:39:49Z)
• A Picture Is Worth a Thousand Words: Exploring Diagram and Video-Based OOP Exercises to Counter LLM Over-Reliance [2.1490831374964587]
  Large language models (LLMs) can effectively solve a range of more complex object-oriented programming (OOP) exercises with text-based specifications. This raises concerns about academic integrity, as students might use these models to complete assignments unethically. We propose an innovative approach to formulating OOP tasks using diagrams and videos, as a way to foster problem-solving and deter students from a copy-and-prompt approach in OOP courses.
  arXiv  Detail & Related papers  (2024-03-13T10:21:29Z)
• Large Language Models as Analogical Reasoners [155.9617224350088]
  Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks. We introduce a new prompting approach, analogical prompting, designed to automatically guide the reasoning process of large language models.
  arXiv  Detail & Related papers  (2023-10-03T00:57:26Z)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists [66.85125112199898]
  We develop a unified language interface for computer vision tasks that abstracts away task-specific design choices. Our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models.
  arXiv  Detail & Related papers  (2023-09-30T14:26:43Z)
• Evaluating ChatGPT and GPT-4 for Visual Programming [20.64766977405438]
  We evaluate two models, ChatGPT (based on GPT-3.5) and GPT-4, in visual programming domains for various scenarios. Our results show that these models perform poorly and struggle to combine spatial, logical, and programming skills crucial for visual programming.
  arXiv  Detail & Related papers  (2023-07-30T22:13:20Z)
• Exploring Large Language Model for Graph Data Understanding in Online Job Recommendations [63.19448893196642]
  We present a novel framework that harnesses the rich contextual information and semantic representations provided by large language models to analyze behavior graphs. By leveraging this capability, our framework enables personalized and accurate job recommendations for individual users.
  arXiv  Detail & Related papers  (2023-07-10T11:29:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.