Andrew McClure
Research Mentor(s): Bing Ye
Mentor Department: Life Sciences Institute, University of Michigan
Authors: Jack McClure, Isabelle Baker, Bing Ye
Session: Session 1 (9:00am – 9:50am)
Presentation Type: Poster 96
Abstract
Methods for categorizing animal behaviors from pre-recorded videos exist such as the original LabGym software, and similar software such as DeepCut and ANY-maze offer live classification. However, none of these offer a feature to predict future behaviors based on previously observed behavior sequences, which would provide more practical applications for behavior analysis software. In order to enable live classification and prediction in the LabGym software, three important features were developed. Firstly, a method for training a configurable transformer through self-supervised learning with a previous output behavior sequence was developed. Then, LabGym was adapted to run in real time. Finally, the transformer was integrated into LabGym’s live classification system in order to provide behavior predictions. The final result is a LabGym mode that can train a transformer with a previously observed behavior sequence in order to enable future behavior predictions based on live video input. Now that behavior categorization software can offer live predictions, the breadth of practical applications greatly expands. For example, the software could be tuned for predicting physical and mental conditions with visual precursors in both animals and humans. Examples include visually predicting specific conditions such as strokes or seizures and general actions such as violence.
