Diego Vargas
Research Mentor(s): Yulia Sevryugina
Mentor Department: Library
Authors: Yulia Sevryugina, Diego Vargas
Session: Session 7 (4:00pm – 4: 50pm)
Presentation Type: Poster 11
Abstract
The rapid advancement of AI has led to its adoption in many areas of our lives. Within academic writing, it can be potentially be utilized as a tool to discover sources, saving time and reducing potential biases. However, it is currently unclear whether or not large language models are efficient at identifying reliable references. This study aimed to investigate the accuracy of references generated by AI to assess the validity of its sources, and understand its limitations. We gathered our data from students across two upper-level undergraduate/graduate courses in Chemistry, who prompted several LLMs to create a 500-word essay containing in-text citations and a bibliography. We recorded reference components including authors, journal names, publication years, page numbers, volume numbers, and DOIs. The references were verified by using the Dimensions Database to categorize them into one of three categories including real and correctly cited, real and incorrectly cited, and fabricated. Of the 483 references, 262 were real and correctly cited, 96 were real and incorrectly cited, and 125 were fabricated. Through additional analysis, significant variations in citation accuracy were uncovered, highlighting the challenges of AI generated references. Based on our findings, it is well identified that solely relying on AI for resources in academic writing is unreliable. As AI continues to evolve, research into the causes of fabrication will be essential to increasing the reliability of the sources it generates. Future studies could explore the effectiveness of more advanced models in generating references.
