Maceo Blumberg
Research Mentor: Geoffrey Siwo
Mentor Department: Learning Health Sciences, Medicine
Author(s): Maceo Blumberg, Geoffery Siwo
Session: Session 1 (9:00 AM – 9:50 AM)
Presentation Type: Poster 65
Abstract
Large language models (LLMs) are increasingly used to support scientific education and research. However, their reliability for bioinformatics tasks involving DNA and RNA sequence analysis remains unclear. Many biological questions require careful, step-by-step reasoning, making it important to evaluate how accurately these models handle sequence-based information. The goal of this project is to examine how different LLMs perform on basic bioinformatics reasoning tasks and to identify common strengths and limitations. In this study, a set of DNA and RNA sequence–based questions was used to evaluate model performance. These questions focused on topics such as sequence interpretation, reasoning about transcription and translation, basic computational biology concepts, and specialized tasks such as gene editing technologies. The same questions were tested across multiple large language models, including newer models (GPT-4 and GPT-5.2) and earlier-generation models (Llama 3, GPT-3.5). Model responses were reviewed for correctness, clarity, and consistency in biological reasoning. Across the tasks tested, GPT-5.2 performed better than earlier models on many DNA analyses and basic bioinformatics problems. It showed strong performance on tasks involving simple sequence processing, probability, genetic reasoning, and CRISPR-related conceptual questions. However, questions that required multi-step algorithms, such as sequence alignment, genome assembly, and graph-based problems, remained difficult. Errors in these areas were common across all models. Overall, this work highlights both the potential and the limitations of large language models in bioinformatics-related tasks. These findings suggest that while LLMs may be useful as learning or support tools, their outputs should be interpreted with caution. Continued evaluation is needed before these models can be reliably integrated into biological research workflows.
