Vikas Tatineni
Pronouns: He/Him
Research Mentor(s): Cynthia Yoonjeong Lee
Research Mentor School/College/Department: Linguistics / LSA
Program: UROPF
Session: Session 7 (4:40pm – 5:30pm)
Authors: Vikas Tatineni, Yoonjeong Lee, McLaren Lindsey, Lynn Yambaye, Jelena Krivokapic
Abstract
Speech communication is multimodal. While the use of body gestures that accompany speech (i.e., co-speech gestures) to highlight important words in an utterance has been extensively studied in English, there is a limited amount of literature about this topic in other languages of different linguistic typologies. To fully understand how language users use multiple modalities in speech communication, it is important to examine languages other than English. The goal of this project is to investigate the ways different speakers use co-speech gestures in their languages. We collected video and kinematic data from a set of Seoul Korean and American English speakers. Speakers from both language groups read a short story in their respective language. The speakers were video recorded, and the movements of various parts of their body (hands, the tongue, lips, head, and eyebrows) were tracked using electromagnetic articulography. We manually labeled co-speech head gestures in the video signal using Adobe Premiere Pro. The linguistically relevant timepoints of the opening and closure of the vocal tract gestures (the lips, tongue tip, and tongue body) and the cooccurring head nods were automatically extracted from the kinematic signal using Matlab. This presentation mainly focuses on how head movements are temporally coordinated with vocal tract gestures and voice pitch during the story reading in Korean speakers. The initial observation suggests that head movements occur frequently when speakers highlight (emphasize) certain words in the story. We plan to compare similarities and differences between individual speaker behaviors within each language and across languages. Results on the precise temporal coordination of speech and co-speech gestures in different languages will be discussed.
