Using CognitIDE to Capture Developers' Cognitive Load via Physiological Activity During Everyday Software Development Tasks

• URL: http://arxiv.org/abs/2503.03537v1
• Date: Wed, 05 Mar 2025 14:19:41 GMT
• Title: Using CognitIDE to Capture Developers' Cognitive Load via Physiological Activity During Everyday Software Development Tasks
• Authors: Fabian Stolp, Charlotte Brandebusemeyer, Franziska Hradilak, Lara Kursawe, Magnus Menger, Franz Sauerwald, Bert Arnrich,
• Abstract summary: We propose a study in which the IntelliJ-based IDE plugin CognitIDE is used to collect, map, and visualize developers' physiological activity data.<n>In a feasibility study, participants completed four simulated everyday working tasks of software developers.<n>CotantIDE could successfully be used for data collection sessions of one hour, which was the most extended duration tested and was well-perceived by those working with it.
• Score: 2.397062421558159
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Integrated development environments (IDE) support developers in a variety of tasks. Unobtrusively capturing developers' cognitive load while working on different programming tasks could help optimize developers' work experience, increase their productivity, and positively impact code quality. In this paper, we propose a study in which the IntelliJ-based IDE plugin CognitIDE is used to collect, map, and visualize software developers' physiological activity data while they are working on various software development tasks. In a feasibility study, participants completed four simulated everyday working tasks of software developers - coding, debugging, code documentation, and email writing - based on Java open source code in the IDE whilst their physiological activity was recorded. Between the tasks, the participants' perceived workload was assessed. Feasibility testing showed that CognitIDE could successfully be used for data collection sessions of one hour, which was the most extended duration tested and was well-perceived by those working with it. Furthermore, the recorded physiological activity indicated higher cognitive load during working tasks compared to baseline recordings. This suggests that cognitive load can be assessed, mapped to code positions, visualized, and discussed with participants in such study setups with CognitIDE. These promising results indicate the usefulness of the plugin for diverse study workflows in a natural IDE environment.

Related papers

• In-IDE Programming Courses: Learning Software Development in a Real-World Setting [5.330251011543498]
  JetBrains recently released the JetBrains Academy plugin, which customizes the IDE for learners.<n>We carried out eight one-hour interviews with students and developers who completed at least one course using the plugin.
  arXiv  Detail & Related papers  (2025-01-29T16:34:22Z)
• Towards Decoding Developer Cognition in the Age of AI Assistants [9.887133861477233]
  We propose a controlled observational study combining physiological measurements (EEG and eye tracking) with interaction data to examine developers' use of AI-assisted programming tools.<n>We will recruit professional developers to complete programming tasks both with and without AI assistance while measuring their cognitive load and task completion time.
  arXiv  Detail & Related papers  (2025-01-05T23:25:21Z)
• Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion? [60.84912551069379]
  We present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Agent is an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage.
  arXiv  Detail & Related papers  (2024-10-02T09:11:10Z)
• How far are AI-powered programming assistants from meeting developers' needs? [17.77734978425295]
  In-IDE AI coding assistant tools (ACATs) like GitHub Copilot have significantly impacted developers' coding habits. We simulate real development scenarios and recruit 27 computer science students to investigate their behavior with three popular ACATs. We find that ACATs generally enhance task completion rates, reduce time, improve code quality, and increase self-perceived productivity.
  arXiv  Detail & Related papers  (2024-04-18T08:51:14Z)
• Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study [72.24266814625685]
  We explore the performance of large language models (LLMs) across the entire software development lifecycle with DevEval. DevEval features four programming languages, multiple domains, high-quality data collection, and carefully designed and verified metrics for each task. Empirical studies show that current LLMs, including GPT-4, fail to solve the challenges presented within DevEval.
  arXiv  Detail & Related papers  (2024-03-13T15:13:44Z)
• Understanding and Evaluating Developer Behaviour in Programming Tasks [0.0]
  In a series of three studies we investigated the specific behaviour of developers solving a specific programming task. We focused on which source code files they visited, how they related pieces of code and knowledge to others and when and how successful they performed code edits.
  arXiv  Detail & Related papers  (2024-03-13T12:46:42Z)
• WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? [83.19032025950986]
  We study the use of large language model-based agents for interacting with software via web browsers. WorkArena is a benchmark of 33 tasks based on the widely-used ServiceNow platform. BrowserGym is an environment for the design and evaluation of such agents.
  arXiv  Detail & Related papers  (2024-03-12T14:58:45Z)
• Improving Testing Behavior by Gamifying IntelliJ [13.086283144520513]
  We introduce IntelliGame, a gamified plugin for the popular IntelliJ Java Integrated Development Environment. IntelliGame rewards developers for positive testing behavior using a multi-level achievement system. A controlled experiment with 49 participants reveals substantial differences in the testing behavior triggered by IntelliGame.
  arXiv  Detail & Related papers  (2023-10-17T11:40:55Z)
• Collaborative, Code-Proximal Dynamic Software Visualization within Code Editors [55.57032418885258]
  This paper introduces the design and proof-of-concept implementation for a software visualization approach that can be embedded into code editors. Our contribution differs from related work in that we use dynamic analysis of a software system's runtime behavior. Our visualization approach enhances common remote pair programming tools and is collaboratively usable by employing shared code cities.
  arXiv  Detail & Related papers  (2023-08-30T06:35:40Z)
• All You Need Is Logs: Improving Code Completion by Learning from Anonymous IDE Usage Logs [55.606644084003094]
  We propose an approach for collecting completion usage logs from the users in an IDE. We use them to train a machine learning based model for ranking completion candidates. Our evaluation shows that using a simple ranking model trained on the past user behavior logs significantly improved code completion experience.
  arXiv  Detail & Related papers  (2022-05-21T23:21:26Z)
• ReACC: A Retrieval-Augmented Code Completion Framework [53.49707123661763]
  We propose a retrieval-augmented code completion framework, leveraging both lexical copying and referring to code with similar semantics by retrieval. We evaluate our approach in the code completion task in Python and Java programming languages, achieving a state-of-the-art performance on CodeXGLUE benchmark.
  arXiv  Detail & Related papers  (2022-03-15T08:25:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.