Ranking hospitals when performance and risk factors are correlated: A simulation-based comparison of risk adjustment approaches for binary outcomes

Autoři: Martin Roessler aff001;  Jochen Schmitt aff001;  Olaf Schoffer aff001
Působiště autorů: Zentrum für Evidenzbasierte Gesundheitsversorgung, Universitätsklinikum und Medizinische Fakultät Carl Gustav Carus an der Technischen Universität Dresden, Dresden, Germany aff001
Vyšlo v časopise: PLoS ONE 14(12)
Kategorie: Research Article
doi: https://doi.org/10.1371/journal.pone.0225844



The conceptualization of hospital quality indicators usually includes some form of risk adjustment to account for hospital differences in case mix. For binary outcome variables like in-hospital mortality, frequently utilized risk adjusted measures include the standardized mortality ratio (SMR), the risk standardized mortality rate (RSMR), and excess risk (ER). All of these measures require the estimation of expected hospital mortality, which is often based on logistic regression models. In this context, an issue that is often neglected is correlation between hospital performance (e.g. care quality) and patient-specific risk factors. The objective of this study was to investigate the impact of such correlation on the adequacy of hospital rankings based on different measures and methods.


Using Monte Carlo simulation, the impact of correlation between hospital care quality and patient-specific risk factors on the adequacy of hospital rankings was assessed for SMR/RSMR, and ER based on logistic regression and random effects logistic regression. As an alternative method, fixed effects logistic regression with Firth correction was considered. The adequacies of the resulting hospital rankings were assessed by the shares of hospitals correctly classified into quintiles according to their true (unobserved) care qualities.


The performance of risk adjustment approaches based on logistic regression and random effects logistic regression declined when correlation between care quality and a risk factor was induced. In contrast, fixed-effects-based estimations proved to be more robust. This was particularly true for fixed-effects-logistic-regression-based ER. In the absence of correlation between risk factors and care quality, all approaches showed similar performance.


Correlation between risk factors and hospital performance may severely bias hospital rankings based on logistic regression and random effects logistic regression. ER based on fixed effects logistic regression with Firth correction should be considered as an alternative approach to assess hospital performance.

Klíčová slova:

Data processing – Death rates – Hospitals – Medical risk factors – Normal distribution – Simulation and modeling – Statistical models


1. Schmitt J, Schoffer O, Walther F, Roessler M, Grählert X, Eberlein-Gonska M, et al. Effectiveness of the IQM peer review procedure to improve in-patient care—a pragmatic cluster randomized controlled trial (IMPRESS): study design and baseline results. Journal of Public Health. 2019. doi: 10.1007/s10389-019-01118-9

2. Krahwinkel W, Schuler E, Liebetrau M, Meier-Hellmann A, Zacher J, Kuhlen R, et al. The effect of peer review on mortality rates. Int J for Qual Health Care. 2016;28(5):594–600. doi: 10.1093/intqhc/mzw072

3. Faber M, Bosch M, Wollersheim H, Leatherman S, Grol R. Public reporting in health care: How do consumers use quality-of-care information?: A systematic review. Med Care. 2009;47(1):1–8. doi: 10.1097/MLR.0b013e3181808bb5 19106724

4. Hafner JM, Williams SC, Koss RG, Tschurtz BA, Schmaltz SP, Loeb JM. The perceived impact of public reporting hospital performance data: interviews with hospital staff. Int J for Qual Health Care. 2011;23(6):697–704. doi: 10.1093/intqhc/mzr056

5. Lichtman JH, Leifheit EC, Wang Y, Goldstein LB. Hospital Quality Metrics:“America’s Best Hospitals” and Outcomes After Ischemic Stroke. J Stroke Cerebrovasc Dis. 2019;28(2):430–434. doi: 10.1016/j.jstrokecerebrovasdis.2018.10.022 30415916

6. Reiter A, Geraedts M, Jäckel W, Fischer B, Veit C, Döbler K. Selection of hospital quality indicators for public disclosure in Germany. Z Evid Fortbild Qual Gesundhwes. 2011;105(1):44–48. doi: 10.1016/j.zefq.2010.12.024 21382604

7. Aylin P, Bottle A, Jen MH, Middleton S, Intelligence F. HSMR mortality indicators. Imperial College Technical Document. 2010.

8. Newman SC. Biostatistical Methods in Epidemiology. New York: Johne Wiley & Sons, INC.; 2001.

9. Normand SLT, Wolf RE, Ayanian JZ, McNeil BJ. Assessing the accuracy of hospital clinical performance measures. Med Decis Making. 2007;27(1):9–20. doi: 10.1177/0272989X06298028 17237448

10. Centers for Medicare & Medicaid Services (CMS). Measure Methodology; 2019. Available from: https://www.cms.gov/Medicare/Quality-Initiatives-Patient-assessment-Instruments/HospitalQualityInits/Measure-Methodology.html.

11. Varewyck M, Goetghebeur E, Eriksson M, Vansteelandt S. On shrinkage and model extrapolation in the evaluation of clinical center performance. Biostatistics. 2014;15(4):651–664. doi: 10.1093/biostatistics/kxu019 24812420

12. Glance LG, Osler T, Shinozaki T. Effect of varying the case mix on the standardized mortality ratio and W statistic: a simulation study. Chest. 2000;117(4):1112–1117. doi: 10.1378/chest.117.4.1112 10767249

13. Kahn JM, Kramer AA, Rubenfeld GD. Transferring critically ill patients out of hospital improves the standardized mortality ratio: a simulation study. Chest. 2007;131(1):68–75. doi: 10.1378/chest.06-0741 17218558

14. Rosenthal GE, Shah A, Way LE, Harper DL. Variations in standardized hospital mortality rates for six common medical diagnoses: implications for profiling hospital quality. Med Care. 1998;36(7):955–964. doi: 10.1097/00005650-199807000-00003 9674614

15. Ryan A, Burgess J, Strawderman R, Dimick J. What is the best way to estimate hospital quality outcomes? A simulation approach. Health Serv Res. 2012;47(4):1699–1718. doi: 10.1111/j.1475-6773.2012.01382.x 22352894

16. Statistische Ämter des Bundes und der Länder; 2019. Available from: https://www.forschungsdatenzentrum.de/en/health/drg.

17. Firth D. Bias reduction of maximum likelihood estimates. Biometrika. 1993;80(1):27–38. doi: 10.1093/biomet/80.1.27

18. Statistische Ämter des Bundes und der Länder. Verzeichnis der Krankenhäuser und Vorsorge- und Rehabilitatsionseinrichtungen in Deutschland; 2019. Available from: https://www.destatis.de/DE/Themen/Gesellschaft-Umwelt/Gesundheit/Krankenhaeuser/_inhalt.html#sprg234206.

19. Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied logistic regression. vol. 398. John Wiley & Sons; 2013.

20. Chen F, Chen S. Injury severities of truck drivers in single-and multi-vehicle accidents on rural highways. Accid Anal Prev. 2011;43(5):1677–1688. doi: 10.1016/j.aap.2011.03.026 21658494

21. Chen F, Chen S, Ma X. Crash Frequency Modeling Using Real-Time Environmental and Traffic Data and Unbalanced Panel Data Models. Int J Environ Res Public Health. 2016;13(6):609. doi: 10.3390/ijerph13060609

22. Chen F, Chen S, Ma X. Analysis of hourly crash likelihood using unbalanced panel data mixed logit model and real-time driving environmental big data. J Safety Res. 2018;65:153–159. doi: 10.1016/j.jsr.2018.02.010 29776524

23. Chen F, Song M, Ma X. Investigation on the Injury Severity of Drivers in Rear-End Collisions Between Cars Using a Random Parameters Bivariate Ordered Probit Model. Int J Environ Res Public Health. 2019;16(14):2632. doi: 10.3390/ijerph16142632

24. Wooldridge JM. Econometric analysis of cross section and panel data. Cambridge: MIT Press; 2010.

25. Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol. 1996;49(12):1373–1379. doi: 10.1016/s0895-4356(96)00236-3 8970487

26. Mansournia MA, Geroldinger A, Greenland S, Heinze G. Separation in logistic regression: causes, consequences, and control. Am J Epidemiol. 2017;187(4):864–870. doi: 10.1093/aje/kwx299

27. Varewyck M, Vansteelandt S, Eriksson M, Goetghebeur E. On the practice of ignoring center-patient interactions in evaluating hospital performance. Stat Med. 2016;35(2):227–238. doi: 10.1002/sim.6634 26303843

28. Glance LG, Dick A, Osler TM, Li Y, Mukamel DB. Impact of changing the statistical methodology on hospital and surgeon ranking: the case of the New York State cardiac surgery report card. Med Care. 2006;44(4):311–319. doi: 10.1097/01.mlr.0000204106.64619.2a 16565631

29. Kalbfleisch JD, Wolfe RA. On monitoring outcomes of medical providers. Stat Biosci. 2013;5(2):286–302. doi: 10.1007/s12561-013-9093-x

30. Moran JL, Solomon PJ, ANZICS Centre for Outcome and Resource Evaluation (CORE) of the Australian and New Zealand Intensive Care Society (ANZICS). Fixed Effects Modelling for Provider Mortality Outcomes: Analysis of the Australia and New Zealand Intensive Care Society (ANZICS) Adult Patient Data-Base. PLoS ONE. 2014;9(7):e102297. doi: 10.1371/journal.pone.0102297 25029164

31. Zeng Q, Wen H, Huang H, Abdel-Aty M. A Bayesian spatial random parameters Tobit model for analyzing crash rates on roadway segments. Accid Anal Prev. 2017;100:37–43. doi: 10.1016/j.aap.2016.12.023 28088033

32. Zeng Q, Wen H, Huang H, Pei X, Wong S. Incorporating temporal correlation into a multivariate random parameters Tobit model for modeling crash rate by injury severity. Transportmetrica A: Transport Science. 2018;14(3):177–191. doi: 10.1080/23249935.2017.1353556

33. Zeng Q, Gu W, Zhang X, Wen H, Lee J, Hao W. Analyzing freeway crash severity using a Bayesian spatial generalized ordered logit model with conditional autoregressive priors. Accid Anal Prev. 2019;127:87–95. doi: 10.1016/j.aap.2019.02.029 30844540

34. Zeng Q, Guo Q, Wong S, Wen H, Huang H, Pei X. Jointly modeling area-level crash rates by severity: a Bayesian multivariate random-parameters spatio-temporal Tobit regression. Transportmetrica A: Transport Science. 2019;15(2):1867–1884. doi: 10.1080/23249935.2019.1652867

35. Zou G. A Modified Poisson Regression Approach to Prospective Studies with Binary Data. Am J Epidemiol. 2004;159(7):702–706. doi: 10.1093/aje/kwh090 15033648

Článek vyšel v časopise


2019 Číslo 12
Nejčtenější tento týden