Audio-visual speech stimuli for the study of lip-reading and multi-sensory integration abilities in hearing-impaired individuals

Authors

  • Maren Stropahl Department of Psychology, Carl von Ossietzky University, Oldenburg, Germany
  • Stefan Debener Department of Psychology, Carl von Ossietzky University, Oldenburg, Germany Cluster of Excellence “Hearing4all”, Oldenburg, Germany

Abstract

Audio-visual integration of speech is frequently investigated with the McGurk effect. Incongruent presentation of auditory and visual syllables may result in the perception of a third syllable, reflecting fusion of visual and auditory information. However, perception of the McGurk effect depends strongly on the stimulus material used, making comparisons across groups and studies difficult. To overcome this limitation we developed a large set of audio-visual speech material, consisting of eight different speakers (4 females and 4 males) and 12 syllable combinations. The quality of the material was evaluated with 24 young and normal-hearing subjects. The McGurk effect was studied in eight adult cochlear implant (CI) users and compared to 24 normal-hearing individuals using a probabilistic model. The comparison confirmed previous reports of stronger audio-visual integration in CI users. The audio-visual material developed in this study will be made freely available.

References

Basu Mallick, D., Magnotti, J.F., and Beauchamp, M.S. (2015). “Variability and stability in the McGurk effect: contributions of participants, stimuli, time, and response type,” Psychon. B. Rev., doi:10.3758/s13423-015-0817-4.

Driver, J. and Noesselt, T. (2008). “Multisensory interplay reveals crossmodal influences on “sensory-specific”brain regions, neural responses, and judgments,” Neuron, 57, 11-23.

Fishman, K.E., Shannon, R.V., and Slattery, W.H. (1997). “Speech recognition as a function of the number of electrodes used in the SPEAK cochlear implant speech processor,” J. Speech Lang. Hear. Res., 40, 1201-1215.

Fu, Q.-J., Shannon, R.V., and Wang, X. (1998). “Effects of noise and spectral resolution on vowel and consonant recognition: Acoustic and electric hearing,” J. Acoust. Soc. Am., 104, 3586-3596.

MacDonald, J. and McGurk, H. (1978). “Visual influences on speech perception processes,” Percept. Psychophys., 24, 253-257.

Magnotti, J.F. and Beauchamp, M.S. (2015). “The noisy encoding of disparity model of the McGurk effect,” Psychon. B. Rev., 22, 701-709. [Source code: http://openwetware.org/wiki/Beauchamp:NED]

McGurk, H. and MacDonald, J. (1976). “Hearing lips and seeing voices,” Nature, 264, 746-748.

Rouger, J., Lagleyre, S., Fraysse, B., Deneve, S., Deguine, O., and Barone, P. (2007). “Evidence that cochlear-implanted deaf patients are better multisensory integrators,” Proc. Nat. Acad. Sci. USA, 104, 7295-7300.

Rouger, J., Fraysse, B., Deguine, O., and Barone, P. (2008). “McGurk effects in cochlear-implanted deaf subjects,” Brain Res., 1188, 87-99.

Sandmann, P., Plotz, K., Hauthal, N., de Vos, M., Schönfeld, R., and Debener, S. (2014). “Rapid bilateral improvement in auditory cortex activity in postlingually deafened adults following cochlear implantation,” Clin. Neurophysiol.. 126, 594-607.

Stropahl, M., Plotz, K., Schönfeld, R., Lenarz, T., Sandmann, P., Yovel, G., De Vos, M., and Debener, S. (2015). “Cross-modal reorganization in cochlear implant users: Auditory cortex contributes to visual face processing,” NeuroImage, 121, 159-170.

Sumby, W. H. and Pollack, I. (1954). “Visual contribution to speech intelligibility in noise,” J. Acoust. Soc.Am., 26, 212-215.

Tremblay, C., Champoux, F., Lepore, F., and Théoret, H. (2009). “Audiovisual fusion and cochlear implant proficiency,” Restor. Neurol. Neuros., 28, 283-291.

van Wassenhove, V., Grant, K.W., and Poeppel, D. (2005). “Visual speech speeds up the neural processing of auditory speech,” Proc. Nat. Acad. Sci. USA, 102, 1181-1186.

Downloads

Published

2015-12-15

How to Cite

Stropahl, M., & Debener, S. (2015). Audio-visual speech stimuli for the study of lip-reading and multi-sensory integration abilities in hearing-impaired individuals. Proceedings of the International Symposium on Auditory and Audiological Research, 5, 437–444. Retrieved from https://proceedings.isaar.eu/index.php/isaarproc/article/view/2015-52

Issue

Vol. 5 (2015): Individual Hearing Loss - Characterization, modelling, compensation strategies

Section

2015/7. Novel methods for behavioral & objective assessment of hearing function

License

Authors who publish with this journal agree to the following terms:

a. Authors retain copyright* and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.

b. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.

c. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).

*From the 2017 issue onward. The Danavox Jubilee Foundation owns the copyright of all articles published in the 1969-2015 issues. However, authors are still allowed to share the work with an acknowledgement of the work's authorship and initial publication in this journal.