Marisa Soto
Pronouns: she/her
Research Mentor(s): Cynthia Yoonjeong Lee
Research Mentor School/College/Department: Linguistics / LSA
Program:
Authors: Marisa Soto, Cynthia Yoonjeong Lee, Sarah Ashby, William Kuch, Alison McGrath, Vikas Tatineni
Session: Session 6: 3:40 pm – 4:30 pm
Poster: 83
Abstract
Co-speech gestures are body movements that co-occur during speech. Associated gestures, such as emphasizing key words using speech gestures that accompany speech. This research focuses on prosody, the rhythmic and melodic aspects of speech that help speakers convey messages effectively. Previous studies extensively examine the relationship between associated co-speech gestures and articulation with respect to grouping words into meaningful chunks—phrases. Since the alignment of speech with the beginning, middle, or end of a prosodic phrase (how speakers group words into meaningful chunks) are not well-explored, this study aims to examine co-speech eyebrow movements and their temporal coordination with speech gestures by looking at both prosodic prominence and phrasing. Eight English speakers, including 4 females and 4 males participated in a multimodal experiment, utilizing electromagnetic articulography (tracking the intra-oral articulation) and simultaneous audio and video recordings. Participants were instructed to read a story as if they were reciting it to a child. Lab members used Adobe Premiere Pro to document any observed co-speech eyebrow movements and categorize these movements by type (such as eyebrow raising or furrow). We also labeled prominent words that were associated with the detected co-speech gestures. The initial labeling done by each labeler was then cross-checked to see if any discrepancies arise from different labelers. The results revealed approximately 134 discrepancies from different labelers. In addition, the project observed enunciated phrases and syllables from the speakers in relation to their phrasal positioning. The purpose of this study provides insight into the temporal coordination of co-speech gestures.
