On the Opportunities of Large Language Models for Programming Process Data

John Edwards; Arto Hellas; Juho Leinonen

doi:10.1145/3716640.3716652

On the Opportunities of Large Language Models for Programming Process Data

John Edwards^*, Arto Hellas, Juho Leinonen

^*Corresponding author for this work

Utah State University

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

2 Downloads (Pure)

Abstract

Computing educators and researchers have long used programming process data to understand how students construct programs and address challenges. Despite its potential, fully automated feedback systems remain underexplored. The emergence of Large Language Models (LLMs) offers new opportunities for analyzing programming data and providing formative feedback. This study explores using LLMs to summarize programming processes and deliver formative feedback. A case study analyzed keystroke-level data from an introductory programming course, processed into code snapshots. Three state-of-the-art LLMs - Claude 3 Opus, GPT-4 Turbo, and LLaMa2 70B Chat - were evaluated for their feedback capabilities. Results show LLMs effectively provide tailored feedback, emphasizing incremental development, algorithmic planning, and code readability. Our findings highlight the potential of combining keystroke data with LLMs to automate formative feedback, showing that the computing education research and practice community is again one step closer to automating formative programming process feedback.

Original language	English
Title of host publication	ACE 2025 - Proceedings of the 27th Australasian Computing Education Conference, Held in conjunction with
Publisher	ACM
Pages	105-113
Number of pages	9
ISBN (Electronic)	9798400714252
DOIs	https://doi.org/10.1145/3716640.3716652
Publication status	Published - 7 Apr 2025
MoE publication type	A4 Conference publication
Event	Australasian Computing Education Conference - Brisbane, Australia Duration: 12 Feb 2025 → 13 Feb 2025 Conference number: 27

Conference

Conference	Australasian Computing Education Conference
Abbreviated title	ACE
Country/Territory	Australia
City	Brisbane
Period	12/02/2025 → 13/02/2025

Keywords

generative AI
keystroke data
large language models
programming process data
programming process feedback
programming process summarization

Access to Document

10.1145/3716640.3716652Licence: CC BY

On the Opportunities of Large Language Models for Programming Process DataFinal published version, 251 KBLicence: CC BY

Cite this

@inproceedings{3a1c58cf3f1742278a947e9d5119704c,

title = "On the Opportunities of Large Language Models for Programming Process Data",

abstract = "Computing educators and researchers have long used programming process data to understand how students construct programs and address challenges. Despite its potential, fully automated feedback systems remain underexplored. The emergence of Large Language Models (LLMs) offers new opportunities for analyzing programming data and providing formative feedback. This study explores using LLMs to summarize programming processes and deliver formative feedback. A case study analyzed keystroke-level data from an introductory programming course, processed into code snapshots. Three state-of-the-art LLMs - Claude 3 Opus, GPT-4 Turbo, and LLaMa2 70B Chat - were evaluated for their feedback capabilities. Results show LLMs effectively provide tailored feedback, emphasizing incremental development, algorithmic planning, and code readability. Our findings highlight the potential of combining keystroke data with LLMs to automate formative feedback, showing that the computing education research and practice community is again one step closer to automating formative programming process feedback.",

keywords = "generative AI, keystroke data, large language models, programming process data, programming process feedback, programming process summarization",

author = "John Edwards and Arto Hellas and Juho Leinonen",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; Australasian Computing Education Conference, ACE ; Conference date: 12-02-2025 Through 13-02-2025",

year = "2025",

month = apr,

day = "7",

doi = "10.1145/3716640.3716652",

language = "English",

pages = "105--113",

booktitle = "ACE 2025 - Proceedings of the 27th Australasian Computing Education Conference, Held in conjunction with",

publisher = "ACM",

address = "United States",

}

On the Opportunities of Large Language Models for Programming Process Data. / Edwards, John; Hellas, Arto ; Leinonen, Juho.
ACE 2025 - Proceedings of the 27th Australasian Computing Education Conference, Held in conjunction with. ACM, 2025. p. 105-113.

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceedings › Scientific › peer-review

TY - GEN

T1 - On the Opportunities of Large Language Models for Programming Process Data

AU - Edwards, John

AU - Hellas, Arto

AU - Leinonen, Juho

N1 - Conference code: 27

PY - 2025/4/7

Y1 - 2025/4/7

N2 - Computing educators and researchers have long used programming process data to understand how students construct programs and address challenges. Despite its potential, fully automated feedback systems remain underexplored. The emergence of Large Language Models (LLMs) offers new opportunities for analyzing programming data and providing formative feedback. This study explores using LLMs to summarize programming processes and deliver formative feedback. A case study analyzed keystroke-level data from an introductory programming course, processed into code snapshots. Three state-of-the-art LLMs - Claude 3 Opus, GPT-4 Turbo, and LLaMa2 70B Chat - were evaluated for their feedback capabilities. Results show LLMs effectively provide tailored feedback, emphasizing incremental development, algorithmic planning, and code readability. Our findings highlight the potential of combining keystroke data with LLMs to automate formative feedback, showing that the computing education research and practice community is again one step closer to automating formative programming process feedback.

AB - Computing educators and researchers have long used programming process data to understand how students construct programs and address challenges. Despite its potential, fully automated feedback systems remain underexplored. The emergence of Large Language Models (LLMs) offers new opportunities for analyzing programming data and providing formative feedback. This study explores using LLMs to summarize programming processes and deliver formative feedback. A case study analyzed keystroke-level data from an introductory programming course, processed into code snapshots. Three state-of-the-art LLMs - Claude 3 Opus, GPT-4 Turbo, and LLaMa2 70B Chat - were evaluated for their feedback capabilities. Results show LLMs effectively provide tailored feedback, emphasizing incremental development, algorithmic planning, and code readability. Our findings highlight the potential of combining keystroke data with LLMs to automate formative feedback, showing that the computing education research and practice community is again one step closer to automating formative programming process feedback.

KW - generative AI

KW - keystroke data

KW - large language models

KW - programming process data

KW - programming process feedback

KW - programming process summarization

UR - http://www.scopus.com/inward/record.url?scp=105007426630&partnerID=8YFLogxK

U2 - 10.1145/3716640.3716652

DO - 10.1145/3716640.3716652

M3 - Conference article in proceedings

AN - SCOPUS:105007426630

SP - 105

EP - 113

BT - ACE 2025 - Proceedings of the 27th Australasian Computing Education Conference, Held in conjunction with

PB - ACM

T2 - Australasian Computing Education Conference

Y2 - 12 February 2025 through 13 February 2025

ER -

On the Opportunities of Large Language Models for Programming Process Data

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Leinonen Juho /AT tot.: Large Language Models for Computing Education

Cite this

On the Opportunities of Large Language Models for Programming Process Data

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Projects

Leinonen Juho /AT tot.: Large Language Models for Computing Education

Cite this