Validation of the Rainbow Model of Integrated Care Measurement Tools (RMIC-MTs) in renal care for patient and care providers

Autoři: Pim P. Valentijn aff001;  Fernando Pereira aff004;  Christina W. Sterner aff005;  Hubertus J. M. Vrijhoef aff001;  Dirk Ruwaard aff002;  Jörgen Hegbrant aff008;  Giovanni F. M. Strippoli aff008
Působiště autorů: Department of Patient and Care, Maastricht University Medical Center, Maastricht, The Netherlands aff001;  Department of Health Services Research, Care and Public Health Research Institute (CAPHRI), Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands aff002;  Integrated Care Evaluation, Essenburgh, Hierden, The Netherlands aff003;  Strategy and Health Economics Office, Diaverum, Madrid, Spain aff004;  Strategy and Health Economics Office, Diaverum, Gothenburg, Sweden aff005;  Department of Family Medicine, Vrije Universiteit Brussel, Brussels, Belgium aff006;  Panaxea, Amsterdam, The Netherlands aff007;  Diaverum Medical Scientific Office, Diaverum Sweden AB, Lund, Sweden aff008;  Sydney School of Public Health, The University of Sydney, Sydney, Australia aff009;  Department of Emergency and Organ Transplantation, University of Bari, Bari, Italy aff010
Vyšlo v časopise: PLoS ONE 14(9)
Kategorie: Research Article
doi: 10.1371/journal.pone.0222593



Integrated service delivery is considered to be an essential condition for improving the management and health outcomes of people with chronic kidney disease (CKD). However, research on the assessment of integrated care by patients and care providers is hindered by the absence of brief, reliable, and valid measurement tools.


The aim of this study was to develop survey instruments for healthcare professionals and patients based on the Rainbow Model of Integrated Care (RMIC), and to evaluate their psychometric properties.


The development process was based on the US Food and Drug Administration guidelines. This included item generation from systematic reviews of existing tools and expert opinion on clarity and content validity, involving renal care providers and chronic kidney patients. A cross-sectional, multi-centre design was used to test for internal consistency and construct validity.


Outpatient clinics in a large renal network.


A sample of 30.788 CKD patients, and 8.914 renal care providers.

Methods and analysis

Both survey instruments were developed using previous qualitative work and published literature. A multidisciplinary expert panel assessed the face and content validity of both instruments and following a pilot study, the psychometric properties of both instruments were explored. Exploratory factor analysis with principal axis factoring and with promax rotation was used to assess the underlying dimensions of both instruments; Cronbach’s alpha was used to determine the internal constancy reliability.


17.512 patients (response rate: 56.9%) and 8.849 care providers (response rate: 69.5%) responded to the questionnaires. Factor analysis of the patient questionnaire yielded three internally consistent (Cronbach’s alpha > 0.7) factors: person-centeredness, clinical coordination, and professional coordination. Factor analysis of the provider questionnaire produced eight internally consistent (Cronbach’s alpha > 0.7) factors: person-centeredness, community centeredness, clinical coordination, professional coordination, organisational coordination, system coordination, technical and cultural competence. As hypothesised, care coordination patient and providers scores significantly correlated with questions about quality of care, treatment involvement, reported health, clinics’ organisational readiness, and external care coordination capacity.


This study provides evidence for the reliability and validity of the RMIC patient and provider questionnaires as generic tools to assess the experience with or perception of integrated renal care delivery. The instruments are recommended in future applications testing test-retest reliability, convergent and predictive validity, and responsiveness.

Klíčová slova:

Medicine and health sciences – Nephrology – Chronic kidney disease – Medical dialysis – Biology and life sciences – Psychology – Psychometrics – Social sciences – Sociology – Culture – Research and analysis methods – Mathematical and statistical techniques – Statistical methods – Factor analysis – Research assessment – Research validity – Research design – Survey research – Questionnaires – Physical sciences – Mathematics – Statistics – Algebra – Linear algebra – Eigenvalues


1. Hill NR, Fatoba ST, Oke JL, Hirst JA, O'Callaghan CA, Lasserson DS, et al. Global Prevalence of Chronic Kidney Disease—A Systematic Review and Meta-Analysis. PLoS One. 2016;11: e0158765. doi: 10.1371/journal.pone.0158765 27383068

2. Stevens LA, Li S, Wang C, Huang C, Becker BN, Bomback AS, et al. Prevalence of CKD and comorbid illness in elderly patients in the United States: results from the Kidney Early Evaluation Program (KEEP). Am J Kidney Dis. 2010;55: S23–33. doi: 10.1053/j.ajkd.2009.09.035 20172445

3. Valentijn PP, Biermann C, Bruijnzeels MA. Value-based integrated (renal) care: setting a development agenda for research and implementation strategies. BMC Health Serv Res. 2016;16: 330-016–1586-0.

4. Kang H, Nembhard HB, Curry W, Ghahramani N, Hwang W. A systems thinking approach to prospective planning of interventions for chronic kidney disease care. Health Systems. 2016.

5. Fishbane S, Hazzan AD, Halinski C, Mathew AT. Challenges and opportunities in late-stage chronic kidney disease. Clin Kidney J. 2015;8: 54–60. doi: 10.1093/ckj/sfu128 25713711

6. Bautista MA, Nurjono M, Lim YW, Dessers E, Vrijhoef HJ. Instruments Measuring Integrated Care: A Systematic Review of Measurement Properties. Milbank Q. 2016;94: 862–917. doi: 10.1111/1468-0009.12233 27995711

7. Uijen AA, Schers HJ, Schellevis FG, van den Bosch WJ. How unique is continuity of care? A review of continuity and related concepts. Fam Pract. 2012;29: 264–271. doi: 10.1093/fampra/cmr104 22045931

8. Kodner DL. All together now: a conceptual exploration of integrated care. Healthc Q. 2009;13 (Special Issue): 6–15. 20057243

9. World Health Organization (WHO). WHO global strategy on integrated people-centred health services 2016–2026. Placing people and communities at the centre of health services. 2015.

10. Valentijn PP, Schepman SM, Opheij W, Bruijnzeels MA. Understanding integrated care: a comprehensive conceptual framework based on the integrative functions of primary care. Int J Integr Care. 2013;13: e010. doi: 10.5334/ijic.886 23687482

11. Boesveld I, Valentijn P, Hitzert M, Hermus M, Franx A, de Vries R, et al. An Approach to measuring Integrated Care within a Maternity Care System: Experiences from the Maternity Care Network Study and the Dutch Birth Centre Study. International journal of integrated care. 2017;17.

12. Angus L, Valentijn PP. From micro to macro: assessing implementation of integrated care in Australia. Aust J Prim Health. 2017.

13. Nurjono M, Bautista MAC, Dessers E, Lim YW, Vrijhoef HJM. Measurement of Integrated Care on the level of Regional Health System in Singapore. Int J Integr Care. 2014;14.

14. Valentijn P, Angus L, Boesveld I, Nurjono M, Ruwaard D, Vrijhoef H. Validating the Rainbow Model of Integrated Care Measurement Tool: results from three pilot studies in the Netherlands, Singapore and Australia. International Journal of Integrated Care. 2017;17. doi: 10.5334/ijic.3104

15. US Food and Drug Administration. Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. 2009.

16. Uijen AA, Heinst CW, Schellevis FG, van den Bosch WJ, van de Laar FA, Terwee CB, et al. Measurement properties of questionnaires measuring continuity of care: a systematic review. PLoS One. 2012;7: e42256. doi: 10.1371/journal.pone.0042256 22860100

17. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007;30: 459–467. doi: 10.1002/nur.20199 17654487

18. Johnson C, Aaronson N, Blazeby J, Bottomley A, Fayers P, Koller M, et al. Guidelines for developing questionnaire modules. 2011: 1–46.

19. Valentijn PP, Vrijhoef HJ, Ruwaard D, de Bont A, Arends RY, Bruijnzeels MA. Exploring the success of an integrated primary care partnership: a longitudinal study of collaboration processes. BMC Health Services Research. 2015;15: 32. doi: 10.1186/s12913-014-0634-x 25609186

20. Costello AB, Osborne JW. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical assessment, research & evaluation. 2005;10: 1–9.

21. Essenburgh Research & Consultancy. Rainbow Model of Integrated Care Measurement Tools (RMIC-MT's) for Patient and Care providers. 2019.

22. Helfrich CD, Li YF, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci. 2009;4: 38-5908-4-38. doi: 10.1186/1748-5908-4-38 19594942

23. Stewart AL. Measuring functioning and well-being: the medical outcomes study approach: duke university Press; 1992.

24. Kline RB. Principles and practice of structural equation modeling: Guilford publications; 2015.

25. Zwart DL, Langelaan M, van de Vooren RC, Kuyvenhoven MM, Kalkman CJ, Verheij TJ, et al. Patient safety culture measurement in general practice. Clinimetric properties of 'SCOPE'. BMC Fam Pract. 2011;12: 117-2296-12-117.

26. Matsunaga M. How to Factor-Analyze Your Data Right: Do's, Don'ts, and How-To's. International journal of psychological research. 2010;3: 97–110.

27. Cerny BA, Kaiser HF. A Study Of A Measure Of Sampling Adequacy For Factor-Analytic Correlation Matrices. Multivariate Behav Res. 1977;12: 43–47. doi: 10.1207/s15327906mbr1201_3 26804143

28. Pallant J. SPSS survival manual: McGraw-Hill Education (UK); 2013.

29. Hair JF, Babin Barry J, Anderson RE, Tatham P L. Multivariate data analysis. Upper Saddle River, NJ: Pearson Prentice Hall; 2005.

30. Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use: Oxford University Press, USA; 2015.

31. van Empel IW, Aarts JW, Cohlen BJ, Huppelschoten DA, Laven JS, Nelen WL, et al. Measuring patient-centredness, the neglected outcome in fertility care: a random multicentre validation study. Hum Reprod. 2010;25: 2516–2526. doi: 10.1093/humrep/deq219 20719811

32. Field A. Discovering statistics using SPSS. Third Edition ed. London: Sage; 2009.

33. Valentijn PP, Pereira FA, Ruospo M, Palmer SC, Hegbrant J, Sterner CW, et al. Person-Centered Integrated Care for Chronic Kidney Disease: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. Clin J Am Soc Nephrol. 2018.

34. Independent Review Board Nijmegen (IRBN). 2015.

35. Orchard CA, King GA, Khalili H, Bezzina MB. Assessment of Interprofessional Team Collaboration Scale (AITCS): development and testing of the instrument. J Contin Educ Health Prof. 2012;32: 58–67. doi: 10.1002/chp.21123 22447712

36. Schroder C, Medves J, Paterson M, Byrnes V, Chapman C, O'Riordan A, et al. Development and pilot testing of the collaborative practice assessment tool. J Interprof Care. 2011;25: 189–195. doi: 10.3109/13561820.2010.532620 21182434

37. Norris J, Carpenter JG, Eaton J, Guo JW, Lassche M, Pett MA, et al. The Development and Validation of the Interprofessional Attitudes Scale: Assessing the Interprofessional Attitudes of Students in the Health Professions. Acad Med. 2015;90: 1394–1400. doi: 10.1097/ACM.0000000000000764 25993280

38. Vanhaecht K, De Witte K, Depreitere R, Van Zelm R, De Bleser L, Proost K, et al. Development and validation of a care process self-evaluation tool. Health Serv Manage Res. 2007;20: 189–202. doi: 10.1258/095148407781395964 17683658

39. Starfield B, Cassady C, Nanda J, Forrest CB, Berk R. Consumer experiences and provider perceptions of the quality of primary care: implications for managed care. J Fam Pract. 1998;46: 216–226. 9519019

40. Dobrow MJ, Paszat L, Golden B, Brown AD, Holowaty E, Orchard MC, et al. Measuring Integration of Cancer Services to Support Performance Improvement: The CSI Survey. Healthc Policy. 2009;5: 35–53. 20676250

41. Uijen AA, Schellevis FG, van den Bosch WJ, Mokkink HG, van Weel C, Schers HJ. Nijmegen Continuity Questionnaire: development and testing of a questionnaire that measures continuity of care. J Clin Epidemiol. 2011;64: 1391–1399. doi: 10.1016/j.jclinepi.2011.03.006 21689904

42. Elwyn G, Barr PJ, Grande SW, Thompson R, Walsh T, Ozanne EM. Developing CollaboRATE: a fast and frugal patient-reported measure of shared decision making in clinical encounters. Patient Educ Couns. 2013;93: 102–107. doi: 10.1016/j.pec.2013.05.009 23768763

43. Campbell HS, Hall AE, Sanson-Fisher RW, Barker D, Turner D, Taylor-Brown J. Development and validation of the Short-Form Survivor Unmet Needs Survey (SF-SUNS). Support Care Cancer. 2014;22: 1071–1079. doi: 10.1007/s00520-013-2061-7 24292016

44. Valentijn PP, Vrijhoef HJ, Ruwaard D, Boesveld I, Arends RY, Bruijnzeels MA. Towards an international taxonomy of integrated primary care: a Delphi consensus approach. BMC Fam Pract. 2015;16: 64-015-0278-x.

45. Singer SJ, Burgers J, Friedberg M, Rosenthal MB, Leape L, Schneider E. Defining and measuring integrated patient care: promoting the next frontier in health care delivery. Med Care Res Rev. 2011;68: 112–127. doi: 10.1177/1077558710371485 20555018

46. Mead N, Bower P. Patient-centred consultations and outcomes in primary care: a review of the literature. Patient Educ Couns. 2002;48: 51–61. 12220750

47. Valentijn PP. Rainbow of Chaos: A study into the Theory and Practice of Integrated Primary Care: Pim P. Valentijn, [S.l.: s.n.], 2015 (Print Service Ede), pp. 195, Doctoral Thesis Tilburg University, The Netherlands, ISBN: 978-94-91602-40-5. Int J Integr Care. 2016;16: 3.

48. Pimouguet C, Le Goff M, Thiebaut R, Dartigues JF, Helmer C. Effectiveness of disease-management programs for improving diabetes care: a meta-analysis. CMAJ. 2011;183: E115–27. doi: 10.1503/cmaj.091786 21149524

49. Renders CM, Valk GD, Griffin S, Wagner EH, Eijk JT, Assendelft WJ. Interventions to improve the management of diabetes mellitus in primary care, outpatient and community settings. Cochrane Database Syst Rev. 2001;(1): CD001481. doi: 10.1002/14651858.CD001481 11279717

50. Gonseth J, Guallar-Castillon P, Banegas JR, Rodriguez-Artalejo F. The effectiveness of disease management programmes in reducing hospital re-admission in older patients with heart failure: a systematic review and meta-analysis of published reports. Eur Heart J. 2004;25: 1570–1595. doi: 10.1016/j.ehj.2004.04.022 15351157

51. Roccaforte R, Demers C, Baldassarre F, Teo KK, Yusuf S. Effectiveness of comprehensive disease management programmes in improving clinical outcomes in heart failure patients. A meta-analysis. Eur J Heart Fail. 2005;7: 1133–1144. doi: 10.1016/j.ejheart.2005.08.005 16198629

52. Badamgarav E, Weingarten SR, Henning JM, Knight K, Hasselblad V, Gano A Jr, et al. Effectiveness of disease management programs in depression: a systematic review. Am J Psychiatry. 2003;160: 2080–2090. doi: 10.1176/appi.ajp.160.12.2080 14638573

53. Neumeyer-Gromen A, Lampert T, Stark K, Kallischnigg G. Disease management programs for depression: a systematic review and meta-analysis of randomized controlled trials. Med Care. 2004;42: 1211–1221. doi: 10.1097/00005650-200412000-00008 15550801

54. Kruis AL, Smidt N, Assendelft WJ, Gussekloo J, Boland MR, Rutten-van Molken M, et al. Integrated disease management interventions for patients with chronic obstructive pulmonary disease. Cochrane Database Syst Rev. 2013;(10):CD009437. CD009437. doi: 10.1002/14651858.CD009437.pub2 24108523

55. Reeve BB, Wyrwich KW, Wu AW, Velikova G, Terwee CB, Snyder CF, et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res. 2013;22: 1889–1905. doi: 10.1007/s11136-012-0344-y 23288613

56. GBD 2016 Healthcare Access and Quality Collaborators. Measuring performance on the Healthcare Access and Quality Index for 195 countries and territories and selected subnational locations: a systematic analysis from the Global Burden of Disease Study 2016. Lancet. 2018;391: 2236–2271. doi: 10.1016/S0140-6736(18)30994-2 29893224

57. Wensing M, Grol R, Smits A. Quality judgements by patients on general practice care: a literature analysis. Soc Sci Med. 1994;38: 45–53. doi: 10.1016/0277-9536(94)90298-4 8146714

58. Coyle J. Understanding dissatisfied users: developing a framework for comprehending criticisms of health care work. J Adv Nurs. 1999;30: 723–731. doi: 10.1046/j.1365-2648.1999.01137.x 10499230

59. Edwards C, Staniszewska S, Crichton N. Investigation of the ways in which patients’ reports of their satisfaction with healthcare are constructed. Sociol Health Illn. 2004;26: 159–183. doi: 10.1111/j.1467-9566.2004.00385.x 15027983

60. Staniszewska SH, Henderson L. Patients’ evaluations of the quality of care: influencing factors and the importance of engagement. J Adv Nurs. 2005;49: 530–537. doi: 10.1111/j.1365-2648.2004.03326.x 15713185

61. Berwick DM, James B, Coye MJ. Connections between quality measurement and improvement. Med Care. 2003;41: I30–8. doi: 10.1097/00005650-200301001-00004 12544814

62. Coulter A, Ellins J. Effectiveness of strategies for informing, educating, and involving patients. BMJ. 2007;335: 24–27. doi: 10.1136/bmj.39246.581169.80 17615222

Článek vyšel v časopise


2019 Číslo 9

Nejčtenější v tomto čísle

Tomuto tématu se dále věnují…


Zvyšte si kvalifikaci online z pohodlí domova

Ulcerative colitis_muž_břicho_střeva
Ulcerózní kolitida
nový kurz

Blokátory angiotenzinových receptorů (sartany)
Autoři: MUDr. Jiří Krupička, Ph.D.

Antiseptika a prevence ve stomatologii
Autoři: MUDr. Ladislav Korábek, CSc., MBA

Citikolin v neuroprotekci a neuroregeneraci: od výzkumu do klinické praxe nejen očních lékařů
Autoři: MUDr. Petr Výborný, CSc., FEBO

Zánětlivá bolest zad a axiální spondylartritida – Diagnostika a referenční strategie
Autoři: MUDr. Monika Gregová, Ph.D., MUDr. Kristýna Bubová

Všechny kurzy