Are we prepared? The development of performance indicators for public health emergency preparedness using a modified Delphi approach

Autoři: Yasmin Khan aff001;  Adalsteinn D. Brown aff004;  Anna R. Gagliardi aff003;  Tracey O’Sullivan aff005;  Sara Lacarte aff001;  Bonnie Henry aff006;  Brian Schwartz aff001
Působiště autorů: Public Health Ontario, Toronto, Ontario, Canada aff001;  Department of Medicine, Division of Emergency Medicine, University of Toronto, Toronto, Ontario, Canada aff002;  University Health Network, Toronto, Ontario, Canada aff003;  Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada aff004;  Faculty of Health Sciences, University of Ottawa, Ottawa, Ontario, Canada aff005;  Office of the Provincial Health Officer, Ministry of Health, Government of British Columbia, Victoria, British Columbia, Canada aff006
Vyšlo v časopise: PLoS ONE 14(12)
Kategorie: Research Article



Disasters and emergencies from infectious diseases, extreme weather and anthropogenic events are increasingly common. While risks vary for different communities, disaster and emergency preparedness is recognized as essential for all nation-states. Evidence to inform measurement of preparedness is lacking. The objective of this study was to identify and define a set of public health emergency preparedness (PHEP) indicators to advance performance measurement for local/regional public health agencies.


A three-round modified Delphi technique was employed to develop indicators for PHEP. The study was conducted in Canada with a national panel of 33 experts and completed in 2018. A list of indicators was derived from the literature. Indicators were rated by importance and actionability until achieving consensus.


The scoping review resulted in 62 indicators being included for rating by the panel. Panel feedback provided refinements to indicators and suggestions for new indicators. In total, 76 indicators were proposed for rating across all three rounds; of these, 67 were considered to be important and actionable PHEP indicators.


This study developed an indicator set of 67 PHEP indicators, aligned with a PHEP framework for resilience. The 67 indicators represent important and actionable dimensions of PHEP practice in Canada that can be used by local/regional public health agencies and validated in other jurisdictions to assess readiness and measure improvement in their critical role of protecting community health.

Klíčová slova:

Canada – Critical care and emergency medicine – Disease surveillance – Exercise – Health services administration and management – Health systems strengthening – Public and occupational health – Public policy


1. WHO Ebola Response Team. Ebola Virus Disease in West Africa—The First 9 Months of the Epidemic and Forward Projections. N Engl J Med. 2014;371: 1481–1495. doi: 10.1056/NEJMoa1411100 25244186

2. Costich JF, Scutchfield FD. Public health preparedness and response capacity inventory validity study. J Public Health Manag Pract. 2004;10: 225–233. doi: 10.1097/00124784-200405000-00006 15253518

3. Public Health Agency of Canada (PHAC). Learning from SARS: Renewal of public health in Canada | Leçons de la crise du SRAS: Renouvellement de la santé publique au Canada. 2003;0-662-34984-9.: 1–22.

4. World Health Organization. A strategic framework for emergency preparedness. 2017. Available from:

5. Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390: 2602–2604. doi: 10.1016/S0140-6736(17)31267-9 28622953

6. United Nations Office for Disaster Risk Reduction (UNISDR). Sendai framework for disaster risk reduction 2015–2030. 2015. Available from:

7. NHS Institute for Innovation and Improvement and the Association of Public Health Observatories. The good indicators guide: understanding how to use and choose indicators. June 1, 2007.

8. Canadian Institute for Health Information (CIHI). Health Indicators 2013. 2013.

9. Veillard J, Huynh T, Ardal S, Kadandale S, Klazinga NS, Brown AD. Making health system performance measurement useful to policy makers: aligning strategies, measurement and local health system accountability in Ontario. Healthcare Policy. 2010;5: 49–65. 21286268

10. Levesque JF, Sutherland K. What role does performance information play in securing improvement in healthcare? A conceptual framework for levers of change. BMJ Open. 2017;7.

11. National Association of County and City Health Officials (NACCHO). National Public Health Performance Standards (NPHPS). 2013.

12. Corso L, Lenaway D, Beitsch L, Landrum L, Deutsch H. National public health performance standards: driving quality improvement in public health systems. J Public Health Manag Pract. 2010;16: 19–23. doi: 10.1097/PHH.0b013e3181c02800 20009640

13. McCabe OL, Barnett DJ, Taylor HG, Links JM. Ready, willing, and able: a framework for improving the public health emergency preparedness system. Disaster Med Public. 2010;4: 161–168.

14. Nelson C, Lurie N, Wasserman J. Assessing public health emergency preparedness: concepts, tools, and challenges. Annu Rev Public Health. 2007;28: 1–18. doi: 10.1146/annurev.publhealth.28.021406.144054 17129174

15. Stoto M. Measuring and assessing public health emergency preparedness. J Public Health Manag Pract. 2013;19: S16–21. doi: 10.1097/PHH.0b013e318294b0e3 23903388

16. World Health Organization. IHR core capacity monitoring framework: checklist and indicators for monitoring progress in the development of IHR core capacities in state parties. 2013. Available from:

17. World Health Organization. Joint external evaluation tool: International Health Regulations (2005). 2016. Available from:;jsessionid=8107D04E9BB61D4C9F4D19F2C08061E9?sequence=1

18. National Health Security Preparedness Index (NHSPI). Explore the index release. 2018. Available from:

19. National Association of County and City Health Officials (NACCHO). Project public health ready criteria: version 8.1. 2016.

20. Toner E, Shearer M, Sell T, et al., Centers for Disease Control and Prevention, John Hopkins Center for Health Security. Health sector resilience checklist for high-consequence infectious diseases-informed by the domestic US Ebola response. 2017. Available from:

21. Toner ES, McGinty M, Schoch-Spana M, Rose DA, Watson M, Echols E, et al. A community checklist for health sector resilience informed by Hurricane Sandy. Health Secur. 2017;15: 53–69. doi: 10.1089/hs.2016.0079 28192055

22. Marcozzi DE, Lurie N. Measuring healthcare preparedness: an all-hazards approach. Isr J Health Policy Res. 2012;1: 42. doi: 10.1186/2045-4015-1-42 23098101

23. Adini B, Goldberg A, Cohen R, Laor D, Bar-Dayan Y. Evidence-based support for the all-hazards approach to emergency preparedness. Isr J Health Policy Res. 2012;1: 40–46. doi: 10.1186/2045-4015-1-40 23098065

24. Emergency Management Policy Directorate, Public Safety Canada. An Emergency Management Framework for Canada, Second Edition. 2011.

25. Public Health Agency of Canada. Lessons Learned Review: Public Health Agency of Canada and Health Canada Response to the 2009 H1N1 Pandemic. 2010.

26. Khan Y, O'Sullivan T, Brown AD, Tracey S, Gibson J, Généreux M, et al. Public health emergency preparedness: A framework to promote resilience. BMC Public Health. 2018;18.

27. World Health Organization Regional Office for Europe. Strengthening health system emergency preparedness: toolkit for assessing health system capacity for crisis management. 2012. Available from:

28. Barnett DJ, Balicer RD, Thompson CB, Storey JD, Omer SB, Semon NL, et al. Assessment of local public health workers' willingness to respond to pandemic influenza through application of the extended parallel process model. PLoS ONE. 2009;4: e6365. doi: 10.1371/journal.pone.0006365 19629188

29. Dorn BC, Savoia E, Testa MA, Stoto MA, Marcus LJ. Development of a survey instrument to measure connectivity to evaluate national public health preparedness and response performance. Public Health Rep. 2007;122: 329–338. doi: 10.1177/003335490712200306 17518304

30. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31: 684–706. doi: 10.1177/0269216317690685 28190381

31. Keeney S, Hasson F, McKenna H. Delphi technique in nursing and health research. First Edition ed. Ames, Iowa: Wiley-Blackwell; 2011.

32. Canadian Institutes of Health Research. Guide to knowledge translation planning at CIHR: integrated and end-of-grant approaches. 2012. Available from:

33. Statistics Canada. Health regions and peer groups. 2015.

34. Quinn PM. Chapter 5. Designing qualitative studies. In: Anonymous Qualitative research and evaluation methods. Thousand Oaks, CA: SAGE Publications; 2015. pp. 243.

35. Mcilfatrick S, Keeney S. Identifying cancer nursing research priorities using the Delphi technique. Journal of Advanced Nursing. 2003;42: 629–636. doi: 10.1046/j.1365-2648.2003.02666.x 12787236

36. Gagliardi A, Simunovic M, Langer B, Stern H, Brown AD. Development of quality indicators for colorectal cancer surgery, using a 3-step modified Delphi approach. Can J Surg. 2005;48: 441–452. 16417050

37. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5: 69. doi: 10.1186/1748-5908-5-69 20854677

38. Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136: 57–65. doi: 10.1016/j.puhe.2015.10.027 26993202

39. Guest G, MacQueen K, Namey E. Applied Thematic Analysis. First ed. Thousand Oaks, CA: Sage; 2012.

40. Seyedin H, Zaboli R, Ravaghi H. Major incident experience and preparedness in a developing country. Disaster Med Public. 2013;7: 313–318.

41. European Centre for Disease Prevention and Control. Public health emergency preparedness—core competencies for EU member states. 2017. Available from:

Článek vyšel v časopise


2019 Číslo 12
Nejčtenější tento týden