Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex

Research output: Contribution to journalArticle

Standard

Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex. / Salmi, Juha; Koistinen, Olli-Pekka; Glerean, Enrico; Jylänki, Pasi; Vehtari, Aki; Jääskeläinen, Iiro; Mäkelä, Sasu; Nummenmaa, Lauri; Nummi-Kuisma, Katarina; Nummi, Ilari; Sams, Mikko.

In: NeuroImage, Vol. 157, 15.08.2017, p. 108-117.

Research output: Contribution to journalArticle

Harvard

APA

Vancouver

Author

Bibtex - Download

@article{e7bf4c726f7e4b489404d2d179724acf,
title = "Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex",
abstract = "During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively.",
keywords = "Audiovisual, fMRI, Multi-voxel pattern analysis, Music, Speech",
author = "Juha Salmi and Olli-Pekka Koistinen and Enrico Glerean and Pasi Jyl{\"a}nki and Aki Vehtari and Iiro J{\"a}{\"a}skel{\"a}inen and Sasu M{\"a}kel{\"a} and Lauri Nummenmaa and Katarina Nummi-Kuisma and Ilari Nummi and Mikko Sams",
year = "2017",
month = "8",
day = "15",
doi = "10.1016/j.neuroimage.2016.12.005",
language = "English",
volume = "157",
pages = "108--117",
journal = "NeuroImage",
issn = "1053-8119",

}

RIS - Download

TY - JOUR

T1 - Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex

AU - Salmi, Juha

AU - Koistinen, Olli-Pekka

AU - Glerean, Enrico

AU - Jylänki, Pasi

AU - Vehtari, Aki

AU - Jääskeläinen, Iiro

AU - Mäkelä, Sasu

AU - Nummenmaa, Lauri

AU - Nummi-Kuisma, Katarina

AU - Nummi, Ilari

AU - Sams, Mikko

PY - 2017/8/15

Y1 - 2017/8/15

N2 - During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively.

AB - During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively.

KW - Audiovisual

KW - fMRI

KW - Multi-voxel pattern analysis

KW - Music

KW - Speech

UR - https://doi.org/10.1016/j.neuroimage.2016.12.005

U2 - 10.1016/j.neuroimage.2016.12.005

DO - 10.1016/j.neuroimage.2016.12.005

M3 - Article

VL - 157

SP - 108

EP - 117

JO - NeuroImage

JF - NeuroImage

SN - 1053-8119

ER -

ID: 9618416