Feature-based audiovisual speech integration of multiple streams
Keywords:
Audiovisual, Speech perception, McGurk effect, Perceived consonant orderAbstract
Speech perception often involves the integration of auditory and visual information. This is shown in the McGurk effect, in which a visual utterance, e.g., /ipi/, dubbed onto an acoustic utterance, e.g., /iki/, produces a combination percept, e.g., /ipki/. However, it is still unclear how phonetic features are integrated audiovisually. Here, we studied audiovisual speech perception by decomposing the auditory component of McGurk combinations into two streams. We show that auditory /i_i/, where the underscore indicates an intersyllabic silence, dubbed onto visual /ipi/ produce a strong illusion of hearing /ipi/. We also show that adding an acoustic release burst to /i_i/ creates a percept of /iki/. An auditory continuum was created with stepwise temporal alignments of the release burst and /i_i/. When dubbed onto /ipi/, this continuum was perceived mostly as a visually driven response /ipi/ when the burst overlapped with either acoustic vowel. Other temporal alignments frequently produced combination responses. Mostly /ikpi/ combinations were obtained when the burst was closer to the initial vowel, and reverse /ipki/ responses when it was closer to the final vowel. These results are indicative of feature-based audiovisual integration where burst and aspiration are sufficient cues for the consonant /k/, while the perception of /p/ depends on place information in the visual stream.
References
Colin, C., Radeau, M., Deltenre, P., Demolin, D., and Soquet, A. (2002). “The role of sound intensity and stop-consonant voicing on McGurk fusions and combinations.” Eur. J. Cogn. Psychol., 14, 475-491.
Green, K. P., and Norrix, L. W. (1997). “Acoustic cues to place of articulation and the McGurk effect: the role of release bursts, aspiration, and formant transitions.” J. Speech Lang. Hear. R., 40, 646-665.
Hampson, M., Guenther, F. H., Cohen, M. A., and Nieto-Castanon, A. (2003). “Changes in the McGurk Effect across phonetic contexts.” Boston University Center for Adaptive Systems and Department of Cognitive and Neural Systems.
Massaro, D. W., and Cohen, M. M. (1993). “Perceiving asynchronous bimodal speech in consonant-vowel and vowel syllables.” Speech Commun., 13, 127-134.
McGurk, H., & MacDonald, J. (1976). “Hearing lips and seeing voices”. Nature, 264, 746-748.
Soto-Faraco, S., and Alsius, A. (2009). “Deconstructing the McGurk–MacDonald illusion.” J. Exp. Psychol. Hum. Percept. Perform., 35(2), 580.
Sumby, W. H., and Pollack, I. (1954). “Visual contribution to speech intelligibility in noise.” J. Acoust. Soc. Am., 26, 212-215.
Additional Files
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:
a. Authors retain copyright* and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
b. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
c. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
*From the 2017 issue onward. The Danavox Jubilee Foundation owns the copyright of all articles published in the 1969-2015 issues. However, authors are still allowed to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
