Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
article has not abstract
Published in the journal:
. PLoS Med 11(2): e32767. doi:10.1371/journal.pmed.1001603
Category: Policy Forum
article has not abstract
The current difficulties in keeping systematic reviews up to date leads to considerable inaccuracy, hampering the translation of knowledge into action.
Incremental advances in conventional review updating are unlikely to lead to substantial improvements in review currency. A new approach is needed.
We propose living systematic review as a contribution to evidence synthesis that combines currency with rigour to enhance the accuracy and utility of health evidence.
Living systematic reviews are high quality, up-to-date online summaries of health research, updated as new research becomes available, and enabled by improved production efficiency and adherence to the norms of scholarly communication.
Together with innovations in primary research reporting and the creation and use of evidence in health systems, living systematic review contributes to an emerging evidence ecosystem.
The Bridge from Evidence to Practice
Health research promises societal benefit by making better health possible. However, there has always been a gap between research findings (what is known) and health care practice (what is done), described as the “evidence-practice” or “know-do” gap . The reasons for this gap are complex , but it is clear that synthesising the complex, incomplete, and at times conflicting findings of biomedical research into forms that can readily inform health decision making is an essential component of the bridge from “knowing” to “doing.”
Systematic reviews (SRs) and meta-analyses have provided incalculable benefit for human health by contributing to the bridge from knowing to doing, but this benefit is limited by characteristics of the current SR enterprise . The methods of SR and meta-analysis are well developed –, but less progress has been achieved with the other essential component of accuracy—currency. The time from the date of the last search to SR publication is commonly over a year , and in an analysis of the time taken for primary study results to be incorporated into an SR, the median time from primary study publication to SR publication ranged from 2.5 to 6.5 years (Figure 1) . Once published, only a minority of reviews are updated within 2 years of publication , and this inability to maintain currency leads to significant inaccuracy. By 2 years post-publication 23% of SRs that have not been updated will have failed to incorporate new evidence that would substantively change conclusions about the effectiveness or harms of therapies .
Current approaches to updating SRs focus on detecting SRs most in need of updating ,. While these methods contribute to the currency of SRs, they cannot adequately reduce the inaccuracy caused by out-of-date SRs. It is often difficult to assemble an authorship team to complete prioritised updates; publication of updates takes many months, during which time the SR remains out of date and therefore potentially inaccurate; and many SRs are not designated as high priority and therefore remain out of date and inaccurate. Despite the availability of these methods and the concerted efforts of many individuals, much of the global corpus of SRs remains out of date.
Incremental advances in traditional SR updating is unlikely to lead to substantial and sustained improvements in the currency of SRs in the context of exponential growth in primary research and SRs . In order to address this considerable source of SR inaccuracy, and produce evidence summaries that are both methodologically rigorous and up to date, a new approach is needed. This innovation has previously been unachievable because rigorous reviews are demanding of time and resources, and “up to date” required rapid processes, which limited the feasibility of rigorous methods. New technologies create the opportunity to resolve this trade-off and enable answers to health questions that are both methodologically rigorous and up to date. We have named this new approach to the updating of SR “living systematic review.” Our aim in proposing this approach is to address the challenges facing contemporary evidence synthesis, while retaining the strengths of SR that have been a critical underpinning of knowledge for health over the last quarter century. This approach to SR adds to rather than replaces existing methods and is particularly relevant for reviews of controlled trials in fast moving topic areas.
Living Systematic Reviews
Living systematic reviews are high quality, up-to-date online summaries of health research that are updated as new research becomes available, consistent with the vision of the pioneering Oxford Database of Perinatal Trials to “include a library of trial overviews, which will be updated when new data become available” . There are four fundamental differences between conventional SR and living systematic review: publication format, work processes, author team management, and statistical methods.
The essential difference between living systematic review and conventional SR is the publication format. Instead of a conventional static SR report or update, living systematic reviews are dynamic, persistent, online-only evidence summaries, which are updated rapidly and frequently. The corollary of a living publication is three changes to SR production. First, work processes must be adapted. A literature search strategy is maintained and outputs fed continuously into an SR workflow, including continuous updating of identified studies, assessment of study quality, data extraction, meta-analysis, and SR report. Instead of the intense, sporadic effort of conventional SRs and SR updates, living systematic review requires moderate, ongoing contribution. Second, author team management must be responsive to a continuous workflow, coordinating effort over long periods of time and allowing evolution in the author team, while maintaining institutional memory.
Third, updating meta-analyses entails a reanalysis of data and, as with repeated analysis of accumulating primary trial data, an inflated rate of false-positive findings is likely if statistical tests are repeated naively . Also, effect size estimates can be unstable, especially in the early stages of evidence generation . These issues are relevant to all updates to meta-analyses, but are particularly important for living systematic reviews given the potential frequency of updates. Statistical monitoring of meta-analyses using formal sequential methods can control the risk of spurious findings, while achieving pre-specified power to identify a pre-specified clinically relevant magnitude of effect ,. Sequential methods are controversial in meta-analysis since they are based on testing rather than estimation, and an estimate that is widely disseminated on the basis of a significance test result may be biased. As an alternative, a Bayesian approach provides a natural framework for monitoring accumulating evidence in which prior distributions can be used to reduce the probability of falsely concluding superiority of an intervention in the early stages of the review , and to stabilize the meta-analysis by exploiting external information about the likely degree of statistical heterogeneity across the studies . Furthermore, a Bayesian framework feeds naturally into decision making.
Production of Living Systematic Reviews
In addition to the three essential changes to review production described above, a key enabler of living systematic review is SR production efficiency. Improvements in efficiency can have profound effects on the application of health technologies. For example, a 4-log decrease in the cost of sequencing a human genome, from US$100 million to US$10,000 over 10 years , has catalysed a genomics revolution with profound benefits for health. In contrast, rising methodological expectations have led to an increase in the complexity and cost of SR and production timelines often in excess of 1–2 years . We describe below several recent developments that have the potential to improve dramatically the efficiency of conventional SR and enable the widespread production of living systematic reviews.
Workflow and Collaboration Tools
Despite the availability of some specific tools, the efforts of most SR authors are fragmented across generic word processing, spreadsheet, email, reference management, and statistical analysis tools ,. This fragmentation hampers the production and updating of SRs, undermines the experience and engagement of SR authors, and limits the availability of process data. Growing innovation in tools and platforms  will enable more efficient SR production, but the right incentives and partnerships need to be in place for these innovations to translate into broadly available applications .
Text-mining technologies are currently being developed to improve the efficiency of SR production . While experimental work encompasses many stages of the review process the most refined techniques are currently concerned with study identification . Here, machine learning technologies have the potential to reduce the manual “screening” of titles and abstracts by up to 50% in new reviews  and more than 90% in review updates , greatly enhancing the efficiency of review production. Other initiatives are developing semi-automation technologies to assist with the development of search strategies, assessment of study quality, extraction of data from documents, and production of SR protocols and reports.
Data Repositories and Linked Data
Important health care questions are often the subject of evidence syntheses by multiple independent groups isolated from each other in redundant effort. The value embedded in the process and output of these parallel activities is only partially captured in discrete, static, and unstructured document-based outputs. Efforts to encourage registration of SRs can help minimise unnecessary duplication . Initiatives that enable SR process and output data to be prospectively stored and reused by others are important developments that will reduce redundant effort ,. When these data are stored in structured formats using shared ontologies ,, following W3C formats for linked data (RDF, OWL), unnecessary duplication can be avoided, but opportunities will also arise to draw from and contribute to the rapidly expanding world of linked open data.
Participation and the Crowd
Despite the fact that SRs are important, resource intensive, and time critical, most SRs are conducted by small academic teams, working part-time over many months. Larger authorship groups increase the efficiency of SR production  and the expertise available to each review group , but remain underutilised, particularly for high priority questions in which both the demand for evidence and engagement are high. In clinical and laboratory research, high priority questions are often addressed by collaborations of dozens or hundreds of researchers working together, but similar undertakings do not currently exist in evidence synthesis. Increasing the involvement of end users in SR production improves the outputs of SR –, and “citizen science” approaches in which citation screening is crowd-sourced from a network of non-expert contributors have also been tested . Efforts to identify smaller units of scholarly effort for dissemination and attribution – may be applicable to SR and encourage broader participation. These approaches to participation need to be evaluated in comparison to conventional approaches and to manage the risk of bias associated with contributors' potential competing interests.
Publication of Living Systematic Reviews
The shift to a persistent, dynamic online-only publication format will be enabled by at least two other developments. First is efficient peer and editorial review of a living document. When an ongoing search strategy identifies no new studies for inclusion, the review can be updated with the date of last search without further review. When new studies or data are identified for inclusion, but these make negligible difference to summary estimates and have no effect on review conclusions, these data can be incorporated into the review with a modest form of review (e.g., editorial only). If new studies or data are identified that result in significant changes to summary estimates or the review's conclusions, these should undergo rapid, but nonetheless robust, peer and editorial review. In the latter situations post-publication peer review can contribute to the accuracy of published reviews.
Second, living systematic reviews should be compatible with the norms of scholarly communication. Attribution of contribution to the living publication can follow existing norms, such as ICMJE criteria. Once authors' contributions no longer fulfil criteria for authorship, they can be removed from the author list and acknowledged as former contributors. Citation can also follow existing practices, including version number/date and date accessed. Finally, current practice can also inform listing in bibliographic databases with minor updates appended to an existing entry and major updates listed as a new publication.
Living Systematic Review and a New Evidence Ecosystem
Living systematic review contributes to the translation of knowledge into practice, primarily because of the contribution currency makes to accuracy and utility, but this approach to evidence synthesis also supports, and is enhanced by, associated upstream and downstream innovations.
In the growing deluge of research the noble science of systematic review resembles archeology: academic teams searching for buried artifacts and working tirelessly to reveal their true meaning. The growth in primary research and availability of diverse research outputs—protocols, trial registration, clinical study reports, and individual patient data—will continue to challenge current SR models. New methods are needed to identify datasets relevant to specific health questions and enable synthesis and insight. For example, annotation of research outputs with richer meta-data will increase the efficiency of review-specific search and screening. More importantly, publication and verification of research outputs in structured forms (e.g., using semantic technologies) will transform review-specific quality assessment and data extraction . Living systematic review, together with these upstream innovations, will ensure that the potential rich insights from large datasets such as clinical study reports, individual patient data, and health system “big data” are made available for health decision making in a rigorous, efficient, and timely manner.
Living systematic review also enhances the efficiency and opportunities for knowledge translation. First, living systematic review enables “living knowledge translation,” including living guidelines, standards, policies, and decision support systems. Second, the value inside the “container” of an SR can be unlocked when the associated data are made available, particularly as open access to linked data formats. The availability of these data opens up opportunities for integration with guideline development platforms and clinical decision support systems  to create a new evidence “ecosystem” (Figure 2).
An essential link between health research and societal benefit is the transformation of millions of published research studies into accurate and usable summaries for health decision making. Many individuals and organisations are committed to this task and their efforts have improved the health of our societies, but increasing demand for accurate evidence—methodologically rigorous and up to date—is not being met. We propose living systematic review as a contribution to the methods of evidence synthesis that addresses these challenges by combining currency with rigour to enhance the accuracy and utility of health evidence. Living systematic review involves modifications to review production and publication, enabled by improved production efficiency and adherence to the norms of scholarly communication. The approach is widely applicable and although challenges remain (Table 1), feasible responses to these challenges exist. Together with emerging innovations in the reporting of primary research and in the creation and use of evidence in health systems, living systematic review contributes to a new evidence ecosystem in which health knowledge and practice are efficiently and rigorously entwined.
1. (2005) Bridging the “know-do” gap: meeting on knowledge translation in global health. Geneva: World Health Organization.
2. GrimshawJM, EcclesMP, LavisJN, HillSJ, SquiresJE (2012) Knowledge translation of research findings. Implement Sci 7: 50.
3. Institute of Medicine (2011) Finding what works in health care: standards for systematic reviews. Washington (D.C.): National Academies Press.
4. MoherD, LiberatiA, TetzlaffJ, AltmanDG (2009) Group P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6: e1000097.
5. Methodological Expectations of Cochrane Intervention Reviews (MECIR) (2013) Available: http://www.editorial-unit.cochrane.org/mecir. Accessed 8 November, 2013.
6. SampsonM, ShojaniaKG, GarrittyC, HorsleyT, OcampoM, et al. (2008) Systematic reviews can be produced and published faster. J Clin Epidemiol 61: 531–536.
7. BraggeP, ClavisiO, TurnerT, TavenderE, CollieA, et al. (2011) The global evidence mapping initiative: scoping research in broad topic areas. BMC Med Res Methodol 11: 92.
8. JadadAR, CookDJ, JonesA, KlassenTP, TugwellP, et al. (1998) Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 280: 278–280.
9. ShojaniaKG, SampsonM, AnsariMT, JiJ, DoucetteS, et al. (2007) How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147: 224–233.
10. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, et al.. (2008) When and how to update systematic reviews. Cochrane Database Syst Rev: MR000023.
11. PattanittumP, LaopaiboonM, MoherD, LumbiganonP, NgamjarusC (2012) A comparison of statistical methods for identifying out-of-date systematic reviews. PLoS ONE 7: e48894.
12. BastianH, GlasziouP, ChalmersI (2010) Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 7: e1000326.
13. ChalmersI (1986) Electronic publications for updating controlled trial reviews. Lancet 328: 287.
14. BerkeyCS, MostellerF, LauJ, AntmanEM (1996) Uncertainty of the time of first significance in random effects cumulative meta-analysis. Control Clin Trials 17: 357–371.
15. IoannidisJ, LauJ (2001) Evolution of treatment effects over time: empirical insight from recursive cumulative metaanalyses. Proc Natl Acad Sci U S A 98: 831–836.
16. WetterslevJ, ThorlundK, BrokJ, GluudC (2008) Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol 61: 64–75.
17. HigginsJP, WhiteheadA, SimmondsM (2011) Sequential methods for random-effects meta-analysis. Stat Med 30: 903–921.
18. SpieglehalterD (2004) Incorporating Bayesian ideas into health-care evaluation. Statist Sci 19: 156–174.
19. TurnerRM, DaveyJ, ClarkeMJ, ThompsonSG, HigginsJP (2012) Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol 41: 818–827.
20. Wetterstrand K (2013) DNA Sequencing costs: data from the NHGRI Genome Sequencing Program (GSP). Available: http://www.genome.gov/sequencingcosts. Accessed 11 July 2013.
21. TriccoAC, BrehautJ, ChenMH, MoherD (2008) Following 411 Cochrane protocols to completion: a retrospective cohort study. PLoS ONE 3: e3684.
22. Elliott J (2012) Exploiting innovations in technology to improve the efficiency of review production. 20th Cochrane Colloquium. Auckland.
23. Ciapponi A, Glujovsky D (2012) Survey among Cochrane authors about early stages of systematic reviews. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012 Auckland, New Zealand
24. ElliottJ, MavergamesC, BeckerL, MeerpohlJ, ThomasJ, et al. (2013) The efficient production of high quality evidence reviews is important for the public good. BMJ 346: f846.
25. ThomasJ, McNaughtJ, AnaniadouS (2011) Applications of text mining within systematic reviews. Research Synthesis Methods 2: 1–14.
26. Thomas J, Brunton J (2010) EPPI-Reviewer 4.0: software for research synthesis. London: Social Science Research Unit, Institute of Education.
27. WallaceB, TrikalinosT, LauJ, BrodleyC, SchmidC (2010) Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics 11: 55.
28. WallaceBC, SmallK, BrodleyCE, LauJ, SchmidCH, et al. (2012) Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining. Genet Med 14: 663–669.
29. BoothA, ClarkeM, DooleyG, GhersiD, MoherD, et al. (2013) PROSPERO at one year: an evaluation of its utility. Systematic Reviews 2: 4.
30. (2013) SRDR: Systematic Review Data Repository. Available: http://srdr.ahrq.gov/. Accessed 8 November 2013.
31. (2013) Cochrane Register of Studies. Available: http://www.metaxis.com/CRSSoftwarePortal/Index.asp. Accessed 8 November 2013.
32. (2013) Cochrane Reviews Ontology. Available: http://www.cochrane.org/sites/default/files/cochrane-ontology.html. Accessed 8 November 2013.
33. (2013) OCRe: The Ontology of Clinical Research. Available: http://rctbank.ucsf.edu/home/ocre. Accessed 8 November 2013.
34. McGee R, Willis N, Jones A, Higgins G, Craig J, et al.. (2012) Factors associated with the timely completion of a Cochrane review. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012Auckland, New Zealand
35. SungNS, CrowleyWF, GenelM, SalberP, SandyL, et al. (2003) Central challenges facing the national clinical research enterprise. JAMA 289: 1278–1287.
36. LavisJ, DaviesH, OxmanA, DenisJL, Golden-BiddleK, et al. (2005) Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy 10 Suppl 135–48.
37. ColbyDC, QuinnBC, WilliamsCH, BilheimerLT, GoodellS (2008) Research glut and information famine: making research evidence more useful for policymakers. Health Aff (Millwood) 27: 1177–1182.
38. Noel-Storr A, Struthers C, Huckvale C, Dooley G, McShane R (2012) ALOIS, ScreenToGo and the Cochrane Register of Studies: using mobile technology to engage and involve consumers in the work of the Cochrane Collaboration. In: Proceedings of the 20th Cochrane Colloquium; 30 September–3 October 2012Auckland, New Zealand.
39. What is the Human Variome Project? Nat Genet 39: 423.
40. GiardineB, BorgJ, HiggsDR, PetersonKR, PhilipsenS, et al. (2011) Systematic documentation and analysis of human genetic variation in hemoglobinopathies using the microattribution approach. Nat Genet 43: 295–301.
41. MonsB, van HaagenH, ChichesterC, HoenPB, den DunnenJT, et al. (2011) The value of data. Nat Genet 43: 281–283.
42. US Food and Drug Administration (2013) Study data standards for regulatory submissions: position statement. Available: http://www.fda.gov/ForIndustry/DataStandards/StudyDataStandards/ucm368613.htm. Accessed 8 November 2013.
43. Brandt L, Kristiansen A, Guyatt G, Vandvik P (2012) Providing likeable and understandable guidelines using GRADE in the EMR with direct links to individual patient data. Phase 1. In: Proceedings of the 9th Guidelines International Network Conference; 22–25 August 2012Berlin, Germany.
Článek vyšel v časopise
2014 Číslo 2
- Aktuální legislativní změny týkající se zdravotnických prostředků – přehledné shrnutí v kostce
- Magnosolv a jeho využití v neurologii
- Není statin jako statin aneb praktický přehled rozdílů jednotlivých molekul
- Nedostatek hořčíku se projevuje u stále více lidí
- Antikoagulační léčba u pacientů před operačními výkony
Nejčtenější v tomto čísle
- Obstructive Sleep Apnea and Risk of Cardiovascular Events and All-Cause Mortality: A Decade-Long Historical Cohort Study
- Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
- Physicians as Fundraisers: Medical Philanthropy and the Doctor-Patient Relationship
- Effect of an Educational Toolkit on Quality of Care: A Pragmatic Cluster Randomized Trial
Zvyšte si kvalifikaci online z pohodlí domova
Deprese u dětí a adolescentůnový kurz