Audio-visual speech stimuli for the study of lip-reading and multi-sensory integration abilities in hearing-impaired individuals

  • Maren Stropahl Department of Psychology, Carl von Ossietzky University, Oldenburg, Germany
  • Stefan Debener Department of Psychology, Carl von Ossietzky University, Oldenburg, Germany Cluster of Excellence “Hearing4all”, Oldenburg, Germany


Audio-visual integration of speech is frequently investigated with the McGurk effect. Incongruent presentation of auditory and visual syllables may result in the perception of a third syllable, reflecting fusion of visual and auditory information. However, perception of the McGurk effect depends strongly on the stimulus material used, making comparisons across groups and studies difficult. To overcome this limitation we developed a large set of audio-visual speech material, consisting of eight different speakers (4 females and 4 males) and 12 syllable combinations. The quality of the material was evaluated with 24 young and normal-hearing subjects. The McGurk effect was studied in eight adult cochlear implant (CI) users and compared to 24 normal-hearing individuals using a probabilistic model. The comparison confirmed previous reports of stronger audio-visual integration in CI users. The audio-visual material developed in this study will be made freely available.


Basu Mallick, D., Magnotti, J.F., and Beauchamp, M.S. (2015). “Variability and stability in the McGurk effect: contributions of participants, stimuli, time, and response type,” Psychon. B. Rev., doi:10.3758/s13423-015-0817-4.

Driver, J. and Noesselt, T. (2008). “Multisensory interplay reveals crossmodal influences on “sensory-specific”brain regions, neural responses, and judgments,” Neuron, 57, 11-23.

Fishman, K.E., Shannon, R.V., and Slattery, W.H. (1997). “Speech recognition as a function of the number of electrodes used in the SPEAK cochlear implant speech processor,” J. Speech Lang. Hear. Res., 40, 1201-1215.

Fu, Q.-J., Shannon, R.V., and Wang, X. (1998). “Effects of noise and spectral resolution on vowel and consonant recognition: Acoustic and electric hearing,” J. Acoust. Soc. Am., 104, 3586-3596.

MacDonald, J. and McGurk, H. (1978). “Visual influences on speech perception processes,” Percept. Psychophys., 24, 253-257.

Magnotti, J.F. and Beauchamp, M.S. (2015). “The noisy encoding of disparity model of the McGurk effect,” Psychon. B. Rev., 22, 701-709. [Source code:]

McGurk, H. and MacDonald, J. (1976). “Hearing lips and seeing voices,” Nature, 264, 746-748.

Rouger, J., Lagleyre, S., Fraysse, B., Deneve, S., Deguine, O., and Barone, P. (2007). “Evidence that cochlear-implanted deaf patients are better multisensory integrators,” Proc. Nat. Acad. Sci. USA, 104, 7295-7300.

Rouger, J., Fraysse, B., Deguine, O., and Barone, P. (2008). “McGurk effects in cochlear-implanted deaf subjects,” Brain Res., 1188, 87-99.

Sandmann, P., Plotz, K., Hauthal, N., de Vos, M., Schönfeld, R., and Debener, S. (2014). “Rapid bilateral improvement in auditory cortex activity in postlingually deafened adults following cochlear implantation,” Clin. Neurophysiol.. 126, 594-607.

Stropahl, M., Plotz, K., Schönfeld, R., Lenarz, T., Sandmann, P., Yovel, G., De Vos, M., and Debener, S. (2015). “Cross-modal reorganization in cochlear implant users: Auditory cortex contributes to visual face processing,” NeuroImage, 121, 159-170.

Sumby, W. H. and Pollack, I. (1954). “Visual contribution to speech intelligibility in noise,” J. Acoust. Soc.Am., 26, 212-215.

Tremblay, C., Champoux, F., Lepore, F., and Théoret, H. (2009). “Audiovisual fusion and cochlear implant proficiency,” Restor. Neurol. Neuros., 28, 283-291.

van Wassenhove, V., Grant, K.W., and Poeppel, D. (2005). “Visual speech speeds up the neural processing of auditory speech,” Proc. Nat. Acad. Sci. USA, 102, 1181-1186.

How to Cite
Stropahl, M., & Debener, S. (2015). Audio-visual speech stimuli for the study of lip-reading and multi-sensory integration abilities in hearing-impaired individuals. Proceedings of the International Symposium on Auditory and Audiological Research, 5, 437-444. Retrieved from
2015/7. Novel methods for behavioral & objective assessment of hearing function