Audiovisual integration in speech perception: a multi-stage process

Authors

  • Kasper Eskelund Cognitive Systems, Department of Informatics and Mathematical Modelling, Technical University of Denmark, DK-2800 Lyngby, Denmark
  • Jyrki Tuomainen Speech Hearing and Language Sciences, University College London, UK
  • Tobias Andersen Cognitive Systems, Department of Informatics and Mathematical Modelling, Technical University of Denmark, DK-2800 Lyngby, Denmark

Abstract

Integration of speech signals from ear and eye is a well-known feature of speech perception. This is evidenced by the McGurk illusion in which visual speech alters auditory speech perception and by the advantage observed in auditory speech detection when a visual signal is present. Here we investigate whether the integration of auditory and visual speech observed in these two audiovisual integration effects are specific traits of speech perception. We further ask whether audiovisual integration is undertaken in a single processing stage or multiple processing stages.

References

Barker, J. P., Berthommier, F., and Schwartz, J.-L. (1998). "Is Primitive AV Coherence an Aid to Segment the Scene?" In Proceedings of the International Conference on Auditory-Visual Speech Processing 1998 (Terrigal, Australia).

Bernstein, L., Auer, E. T. J., and Takayanagi, S. (2004)." Auditory speech detection in noise enhanced by lipreading". Speech Communication 44, 5-18

Bregman, A. S. (1990). Auditory scene analysis: the perceptual organization of sound (MIT Press).

Eskelund, K., Tuomainen, J., and Andersen, T. S. (2010). "Multistage audiovisual integration of speech: dissociating identification and detection". Exp. Brain Res. 208, 447–457.

Grant, K. W., and Seitz, P.-F. (2000). "The use of visible speech cues for improving auditory detection of spoken sentences". J. Acoust. Soc. Am. 108, 1197- 1208.

Massaro, D. W. (1998). Perceiving talking faces: from speech perception to a behavioral principle (MIT Press).

McGurk, H., and MacDonald, J. (1976). "Hearing lips and seeing voices". Nature 264, 746–748.

Nahorna, O., Berthommier, F., and Schwartz, J.-L. (2011). "Binding and unbinding in audiovisual speech fusion: Follow-up experiments on a new paradigm". In Proceedings of the International Conference on Auditory-Visual Speech Processing 2011 (Volterra, Italy: Kungliga Tekniska Högskolan, Sweden).

Nahorna, O., Berthommier, F., and Schwartz, J.-L. (2010). "Binding and unbinding in audiovisual speech fusion: Removing the McGurk effect by an incoherent preceding audiovisual context". In Proceedings of the International Conference on Auditory-Visual Speech Processing 2010 (Hakone, Kanagawa, Japan: Kumamoto University, Japan).

Remez, R., Rubin, P., Pisoni, D., and Carrell, T. (1981). "Speech perception without traditional speech cues". Science 212, 947–949.

Additional Files

Published

2011-12-15

How to Cite

Eskelund, K., Tuomainen, J., & Andersen, T. (2011). Audiovisual integration in speech perception: a multi-stage process. Proceedings of the International Symposium on Auditory and Audiological Research, 3, 119–126. Retrieved from https://proceedings.isaar.eu/index.php/isaarproc/article/view/2011-14

Issue

Vol. 3 (2011): Speech Perception and Auditory Disorders

Section

2011/1. Indicators of hearing impairment and measures of speech perception

License

Authors who publish with this journal agree to the following terms:

a. Authors retain copyright* and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.

b. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.

c. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).

*From the 2017 issue onward. The Danavox Jubilee Foundation owns the copyright of all articles published in the 1969-2015 issues. However, authors are still allowed to share the work with an acknowledgement of the work's authorship and initial publication in this journal.