Kavin Goyal
Research Mentor(s): Brian Perron
Mentor Department: Child & Adolescent Data Lab
Authors:
Session: Session 1 (9:00am – 9:50am)
Presentation Type: Poster 80
Abstract
Artificial Intelligence technologies are fundamentally reshaping society, and social work stands to benefit significantly from AI-powered tools. However, given that mistakes in social work contexts can have serious consequences for vulnerable populations, maintaining human oversight is crucial. This project demonstrates the implementation of a human-in-the-loop (HITL) approach through an integrated data pipeline that serves two primary purposes: continuous performance monitoring and gold-standard dataset creation. Our system automates data flow between Large Language Models (LLMs) and human reviewers, eliminating manual data transfer and calculation steps. Human experts review model outputs through a purpose-built graphical interface, simultaneously generating performance metrics for real-time monitoring and creating verified datasets for benchmark testing. This automated integration of human oversight enables both ongoing quality control and the systematic development of evaluation datasets for extraction and classification tasks. Our implementation leverages open-source software to create an accessible interface optimized for non-technical users, streamlining the review process while maintaining rigorous oversight of AI systems in social work applications.
