Which components of the face do humans and machines best speechread?

Abstract

Which components of the face are most informative to speechreaders? We here bring some answers through a series of experiments recently run at the ICP, with the whole face of a French speaker, with his lips only, and with several components of a synthetic face. The synthetic models were controlled by a small set of command parameters directly measured on the same speaker’s face. All visual displays were synchronized to the original acoustic speech, previously degraded at various levels of additive noise.

Speechreaders certainly don’t “read” parameters on a speaker’s face. They rather make use of an holistic set of visual cues. However, it is enlightening to quantify the benefit of “parameter-reading” by means of a parametric model of the face seen by human listeners on the one hand, and with an HMM-based automatic speech recognizer using those parameters as an input on the other hand.

Such parallel studies on speechreading by man and machine require i) a limited and constrained corpus representative of a given language, ii) a special workstation / software to accurately measure reliable parameters, and iii) the design of parametric models of the most important components of the human face. All those tools are here presented. Finally, the most striking results from intelligibility tests run by human perceivers thanks to this material are discussed and compared to visual recognition scores obtained by HMMs trained on various sets of parameters with the same visual database.