Henry Gold
Research Mentor: James Dumlao
Mentor Department: Not Available, Information
Author(s): Henry Gold, Meng Wang, James Zumel Dumlao
Session: Session 2 (10:00 AM – 10:50 AM)
Presentation Type: Poster 83
Abstract
As artificial intelligence tools become increasingly prevalent in higher education, instructors face critical challenges deciding how to manage students’ generative AI usage in their courses. University syllabi are the primary way through which faculty communicate these AI policies, yet little is known about the variation in these policies across different disciplines and course levels. In this project, we collected and manually annotated over 500 syllabi from the University of Michigan to capture AI policy statements and relevant course data. This includes extracting course logistics such as instructional team composition (number of instructors, GSIs, and IAs) and office hour availability, enabling analysis of how instructional staffing and support structures vary across course types. We implemented a consensus coding process, where each syllabus is annotated by two humans and final labels are decided through annotator deliberation. We measure inter-rater reliability to understand which annotation tasks are more subjective, and therefore more difficult to automate using large language models (LLMs). Building on this, we developed a framework using LLMs to automatically extract and classify AI policy information from syllabi at scale. We evaluated the large language model’s performance against our annotations using standard metrics, such as precision, accuracy, recall, and F1 scores, and conducted error analysis to identify where the LLM and our own classification diverge in judgment. Our analysis reveals insights into the factors that explain policy variation. Our work establishes both a methodological approach for analyzing course policies at scale and empirical findings about the landscape of AI in higher education, with implications for understanding how these policies eventually affect student learning.
