Facial configuration and audiovisual integration of speech: a mismatch negativity study

Authors

  • Kasper Eskelund Section for Cognitive Systems, Department of Applied Mathematics and Computer Science, Technical University of Denmark; CHeSS, Oticon Centre for Hearing and Speech Sciences, Technical University of Denmark
  • Laura Frølich Section for Cognitive Systems, Department of Applied Mathematics and Computer Science, Technical University of Denmark
  • Tobias S. Andersen Section for Cognitive Systems, Department of Applied Mathematics and Computer Science, Technical University of Denmark; CHeSS, Oticon Centre for Hearing and Speech Sciences, Technical University of Denmark

Abstract

Visual speech plays a central role in general speech perception. Through audiovisual integration, visual speech may facilitate auditory detection and identification for people with normal hearing in noisy conditions. Further, a visual syllable may alter the auditory phonetic percept, as can be seen in the McGurk illusion. In this study, we investigate the role of the configuration of facial features in perception of audiovisual speech. Face perception is known to be highly sensitive to specific arrangements of facial features. By nature, visual speech perception – and thus bimodal integration of audio- visual speech – relies on information from the talking face. However, visual speech encoding and face perception are known to be functionally separate. Previous behavioral findings have shown that for some speech tokens, audiovisual speech perception is altered when the facial configuration is manipulated, even though the constituent features are unchanged. This suggests a functional dependency between the encoding of audiovisual speech and face perception. Here, we investigate the effect by means of electrophysiology in a mismatch-negativity paradigm. Specifically, we present stimuli that support face perception and stimuli that do not, but only find mismatch negativity indicating audiovisual integration with the former.

References

Blair, R.C., and Karniski, W. (1993). “An alternative method for significance testing of waveform difference potentials,” Psychophysiology, 30, 518-524.

Bruce, V., and Young, A. (1986). “Understanding face recognition,” Br. J. Psychol., 77, 305-327.

Calvert, G.A., and Campbell, R. (2003). “Reading speech from still and moving faces: The neural substrates of visible speech,” J. Cogn. Neurosci., 15, 57-70.

Colin, C. (2002). “Mismatch negativity evoked by the McGurk–MacDonald effect: a pho- netic representation within short-term memory,” Clin. Neurophysiol., 113, 495-506.

Delorme, A., and Makeig, S. (2004). “EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis,” J. Neurosci. Meth., 134, 9-21.

Eskelund, K., Tuomainen, J., and Andersen, T.S. (2010). “Multistage audiovisual integration of speech: dissociating identification and detection,” Exp. Brain Res., 208, 447-457.

Grant, K.W., and Seitz, P.-F. (2000). “The use of visible speech cues for improving auditory detection of spoken sentences,” J. Acoust. Soc. Am., 108, 1197-1208.

McGurk, H., and MacDonald, J. (1976). “Hearing lips and seeing voices,” Nature, 264, 746-748.

Näätänen, R., Gaillard, A.W.K., and Mäntysalo, S. (1978). “Early selective-attention effect on evoked potential reinterpreted,” Acta Psychol., 42, 313-329.

Näätänen, R. (2003). “Mismatch negativity: clinical research and possible applications,” Int. J. Psychophysiol., 48, 179-188.

Rosenblum, L.D., Yakel, D.A., and Green, K.P. (2000). “Face and mouth inversion effects on visual and audiovisual speech perception,” J. Exp. Psychol. Hum. Percept. Perform., 26, 806-819.

Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O.V., Lu, S.-T., and Simola, J. (1991). “Seeing speech: visual information from lip movements modifies activity in the human auditory cortex,” Neurosci. Lett., 127, 141-145.

Sumby, W.H., and Pollack, I. (1954). “Visual contribution to speech intelligibility in noise,” J. Acoust. Soc. Am., 26, 212-215.

Thompson, P. (1980). “Margaret Thatcher: a new illusion,” Perception, 9, 483-484.

Additional Files

Published

2013-12-15

How to Cite

Eskelund, K., Frølich, L., & Andersen, T. S. (2013). Facial configuration and audiovisual integration of speech: a mismatch negativity study. Proceedings of the International Symposium on Auditory and Audiological Research, 4, 437–444. Retrieved from https://proceedings.isaar.eu/index.php/isaarproc/article/view/2013-50

Issue

Vol. 4 (2013): Auditory Plasticity - Listening with the Brain

Section

2013/9. Audiovisual interactions and music-listening habits

License

Authors who publish with this journal agree to the following terms:

a. Authors retain copyright* and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.

b. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.

c. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).

*From the 2017 issue onward. The Danavox Jubilee Foundation owns the copyright of all articles published in the 1969-2015 issues. However, authors are still allowed to share the work with an acknowledgement of the work's authorship and initial publication in this journal.