Ordinal models of audiovisual speech perception
Abstract
Audiovisual information is integrated in speech perception. One manifestation of this is the McGurk illusion in which watching the articulating face alters the auditory phonetic percept. Understanding this phenomenon fully requires a computational model with predictive power. Here, we describe ordinal models that can account for the McGurk illusion. We compare this type of models to the Fuzzy Logical Model of Perception (FLMP) in which the response categories are not ordered. While the FLMP generally fit the data better than the ordinal model it also employs more free parameters in complex experiments when the number of response categories are high as it is for speech perception in general. Testing the predictive power of the models using a form of cross-validation we found that ordinal models perform better than the FLMP. Based on these findings we suggest that ordinal models generally have greater predictive power because they are constrained by a priori information about the adjacency of phonetic categories.
References
Busemeyer, J. R., and Wang, Y. M. (2000). “Model comparisons and model selections based on generalization criterion methodology” Journal of Mathematical Psychology, 44(1), 171-189.
Hill, G. (1977). Algorithm 518: Incomplete Bessel Function I0: The Von Mises Distribution. ACM Transactions on Mathematical Software, 3(3), 279-284.
MacDonald, J., and McGurk, H. (1978). “Visual influences on speech perception processes” Percept. Psychophys., 24, 253-257.
Massaro, D. W. (1998). Perceiving talking faces. Cambridge, Massachusetts: MIT Press.
Massaro, D. W., Cohen, M. M., Campbell, C. S., and Rodriguez, T. (2001). “Bayes factor of model selection validates FLMP” Psychon Bull Rev, 8 (1), 1-17.
McGurk, H., and MacDonald, J. (1976). “Hearing lips and seeing voices” Nature, 264, 746-748.
Myung, I. J., Pitt, M. A., and Kim, W. (2005). “Model Evaluation, Testing and Selection” In K. Lamberts and R. L. Goldstone (Eds.), The Handbook of Cognition. London: Sage Publications.
Pitt, M. A., Myung, I. J., and Zhang, S. B. (2002). “Toward a method of selecting among computational models of cognition” Psychological Review, 109 (3), 472-491.
Schwartz, J. L. (2006). “The 0/0 problem in the fuzzy-logical model of perception” J. Acoust. Soc. Am., 120 (4), 1795-1798.
Sumby, W. H., and Pollack, I. (1954). “Visual contribution to speech intelligibility in noise” J. Acoust. Soc. Am., 26(2), 212-215.
Additional Files
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:
a. Authors retain copyright* and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
b. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
c. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
*From the 2017 issue onward. The Danavox Jubilee Foundation owns the copyright of all articles published in the 1969-2015 issues. However, authors are still allowed to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
