Experiences from Integrating Large Language Model Chatbots into the Classroom

We provided students access to a state-of-the-art large language model (LLM) chatbot through the online materials of three university-level courses. One of the courses focused on software engineering with LLMs, while the two other courses were not directly related to LLMs. The chatbot used OpenAI GPT-4 without additional filters or system prompts. Our results suggest that only a minority of students engage with the chatbot in the courses that do not relate to LLMs. At the same time, unsurprisingly, nearly all students in the LLM-focused course leveraged the chatbot. In all courses, the majority of the chatbot usage came from a few superusers, whereas the majority of the students did not heavily use the chatbot even though it effectively provided free access to OpenAI's GPT-4 model (which would have otherwise required a paid subscription at the time of the study). We observe that in addition to students using the chatbot for course-specific purposes, many use the chatbot for their own purposes. Overall, our results suggest that the worst fears of educators -- all students overrelying on chatbots -- did not materialize. Finally, we discuss potential reasons for low usage, including the need for more tailored and scaffolded chatbot experiences targeted for specific types of use cases.