Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex

Juha Salmi, Olli-Pekka Koistinen, Enrico Glerean, Pasi Jylänki, Aki Vehtari, Iiro Jääskeläinen, Sasu Mäkelä, Lauri Nummenmaa, Katarina Nummi-Kuisma, Ilari Nummi, Mikko Sams

Tutkimustuotos: LehtiartikkeliArticleScientificvertaisarvioitu

5 Sitaatiot (Scopus)
398 Lataukset (Pure)

Abstrakti

During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively.
AlkuperäiskieliEnglanti
Sivut108-117
Sivumäärä10
JulkaisuNeuroImage
Vuosikerta157
DOI - pysyväislinkit
TilaJulkaistu - 15 elok. 2017
OKM-julkaisutyyppiA1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä

Sormenjälki

Sukella tutkimusaiheisiin 'Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex'. Ne muodostavat yhdessä ainutlaatuisen sormenjäljen.

Siteeraa tätä