Transforming assessment of speech in children with cleft palate via online crowdsourcing

Autoři: Anne M. Sescleifer aff001;  Caitlin A. Francoisse aff001;  Janna C. Webber aff002;  Jeffrey D. Rector aff003;  Alexander Y. Lin aff001
Působiště autorů: Division of Plastic Surgery, Saint Louis University School of Medicine, Saint Louis, Missouri, United States of America aff001;  St. Louis Cleft-Craniofacial Center, Division of Pediatric Plastic Surgery, SSM Health Cardinal Glennon Children’s Hospital at SLU, Saint Louis, Missouri, United States of America aff002;  Rector Consulting, San Francisco, California, United States of America aff003
Vyšlo v časopise: PLoS ONE 15(1)
Kategorie: Research Article



Speech intelligibility is fundamental to social interactions and a critical surgical outcome in patients with cleft palate. Online crowdsourcing is a burgeoning technology, with potential to mitigate the burden of limited accessibility to speech-language-pathologists (SLPs). This pilot study investigates the concordance of online crowdsourced evaluations of hypernasality with SLP ratings of children with cleft palate.


Six audio-phrases each from children with cleft palate were assessed by online crowdsourcing using Amazon Mechanical Turk (MTurk), and compared to SLP’s gold-standard hypernasality score on the Pittsburgh Weighted Speech Score (PWSS). Phrases were presented to MTurk crowdsourced lay-raters to assess hypernasality on a Likert scale analogous to the PWSS. The survey included clickable reference audio samples for different levels of hypernasality.


1,088 unique online crowdsourced speech ratings were collected on 16 sentences of 3 children with cleft palate aged 4–8 years, with audio averaging 6.5 years follow-up after cleft palate surgery. Patient 1 crowd-mean was 2.62 (SLP rated 2–3); Patient 2 crowd-mean 2.66 (SLP rated 3); and Patient 3 crowd-mean 1.76 (SLP rated 2). Rounded for consistency with PWSS scale, all patients matched SLP ratings. Different sentences had different accuracies compared to the SLP gold standard scores.


Online crowdsourced ratings of hypernasal speech in children with cleft palate were concordant with SLP ratings, predicting SLP scores in all 3 patients. This novel technology has potential for translation in clinical speech assessments, and may serve as a valuable screening tool for non-experts to identify children requiring further assessment and intervention by a qualified speech language pathology expert.

Klíčová slova:

Cleft palate – Children – Phonemes – Pilot studies – Speech – Surgical and invasive medical procedures – Speech-language pathology – Cleft palate surgery


1. Parker SE, Mai CT, Canfield MA, Rickard R, Wang Y, Meyer RE, et al. Updated National Birth Prevalence estimates for selected birth defects in the United States, 2004–2006. Birth Defects Res A Clin Mol Teratol. 2010;88(12):1008–16. Epub 2010/09/30. doi: 10.1002/bdra.20735 20878909

2. Abbott M. Cleft lip and palate. Pediatrics in review. 2014;35(5):177–181. doi: 10.1542/pir.35-5-177 24790071

3. Standards for approval of cleft palate and craniofacial teams (Standard No. 1.2). American Cleft Palate-Craniofacial Association; 2016.

4. Naran S, Ford M, Losee JE. What's New in Cleft Palate and Velopharyngeal Dysfunction Management? Plastic and reconstructive surgery. 2017;139(6):1343e–55e. Epub 2017/05/26. doi: 10.1097/PRS.0000000000003335 28538580

5. Reilly S, Harper M, Goldfeld S. The demand for speech pathology services for children: Do we need more or just different? Journal of paediatrics and child health. 2016;52(12):1057–61. Epub 2016/09/24. doi: 10.1111/jpc.13318 27662112

6. Sescleifer AM, Francoisse CA, Lin AY. Systematic Review: Online Crowdsourcing to Assess Perceptual Speech Outcomes. The Journal of surgical research. 2018;232:351–64. Epub 2018/11/23. doi: 10.1016/j.jss.2018.06.032 30463741

7. McWilliams BJ PB. Velopharyngeal Incompetence: Audio Seminars in Speech Pathology. Philadelphia: W.B. Saunders, Inc.; 1979.

8. Brunnegard K, Lohmander A, van Doorn J. Untrained listeners' ratings of speech disorders in a group with cleft palate: a comparison with speech and language pathologists' ratings. International journal of language & communication disorders. 2009;44(5):656–74. Epub 2008/09/30. doi: 10.1080/13682820802295203 18821109

9. Lewis KE, Watterson TL, Houghton SM. The influence of listener experience and academic training on ratings of nasality. Journal of communication disorders. 2003;36(1):49–58. Epub 2002/12/21. doi: 10.1016/s0021-9924(02)00134-x 12493637

10. Starr CD, Moller KT, Dawson W, Graham J, Skaar S. Speech ratings by speech clinicians, parents and children. The Cleft palate journal. 1984;21(4):286–92. Epub 1984/10/01. 6595084

11. Tonz M, Schmid I, Graf M, Mischler-Heeb R, Weissen J, Kaiser G. Blinded speech evaluation following pharyngeal flap surgery by speech pathologists and lay people in children with cleft palate. Folia phoniatrica et logopaedica: official organ of the International Association of Logopedics and Phoniatrics (IALP). 2002;54(6):288–95. Epub 2002/11/06. doi: 10.1159/000066153 12417799

12. Bagnall AD, David DJ. Speech results of cleft palate surgery: two methods of assessment. British journal of plastic surgery. 1988;41(5):488–95. Epub 1988/09/01 doi: 10.1016/0007-1226(88)90005-7 3179594

13. Witt PD, Berry LA, Marsh JL, Grames LM, Pilgram TK. Speech outcome following palatoplasty in primary school children: do lay peer observers agree with speech pathologists? Plastic and reconstructive surgery. 1996;98(6):958–65; discussion 66–70. Epub 1996/11/01. doi: 10.1097/00006534-199611000-00005 8911465

14. McNaney R OM, Richardson D, et al., editor Speeching: Mobile Crowdsourced Speech Assessment to Support Self-Monitoring and Management for People with Parkinson’s. CHI Conference on Human Factors in Computing Systems; 2016; New York, NY.

15. Lohmander A, Olsson M. Methodology for perceptual assessment of speech in patients with cleft palate: a critical review of the literature. The Cleft palate-craniofacial journal: official publication of the American Cleft Palate-Craniofacial Association. 2004;41(1):64–70. Epub 2003/12/31. doi: 10.1597/02-136 14697067

Článek vyšel v časopise


2020 Číslo 1
Nejčtenější tento týden