Measuring sustainability of seed-funded earth science informatics projects

Autoři: Leslie Hsu aff001;  Vivian B. Hutchison aff001;  Madison L. Langseth aff001
Působiště autorů: U.S. Geological Survey, Science Analytics and Synthesis, Denver, Colorado, United States of America aff001
Vyšlo v časopise: PLoS ONE 14(10)
Kategorie: Research Article
doi: 10.1371/journal.pone.0222807


Short term funding is a common funding model for informatics projects. Funders are interested in maximizing the sustainability and accessibility of the outputs, but there are no commonly accepted practices to do so in the Earth sciences informatics field. We constructed and applied a framework for sustainability drawing from other disciplines that have more published work focusing on sustainability of projects. This framework had seven sustainability influences (outputs modified, code repository used, champion present, workforce stability, support from other organizations, collaboration/partnership, and integration with policy), and three ways of defining sustainability (at the individual-, organization-, and community-level). Using this framework, we evaluated outputs of projects funded by the U.S. Geological Survey’s Community for Data Integration (CDI). We found that the various outputs are widely accessible, but not necessarily sustained or maintained. Projects with most sustainability influences often became institutionalized and met required needs of the community. Even if proposed outputs were not delivered or sustained, knowledge of lessons learned could be spread to build community capacity in a topic, which is another type of sustainability. We conclude by summarizing lessons for individuals applying for short-term funding, and for organizations managing programs that provide such funding, for maximizing sustainability of project outcomes.

Klíčová slova:

Computer software – Data management – Earth sciences – Metadata – Research grants – Science policy – Software tools – Sustainability science


1. Maron N. A guide to the best revenue models and funding sources for your digital resources. Ithaka S+ R. 2014, Available from:

2. Signell RP, Barker C, Dalyander P, Hunt C, Knudsen R, Ring K, et al. Evaluating a new open-source, standards-based framework for web portal development in the geosciences: ScienceBase 2016. Available from:

3. Geoweaver: a web-based prototype system for managing compound geospatial workflows of large-scale distributed deep networks [Internet].[cited 2019 Aug 15]. Available from:

4. Sediment Experimentalists Network [Internet].[cited 2019 Aug 15]. Available from:

5. Gesing S, Wilkins-Diehr N, Dahan M, Lawrence K, Zentner M, Pierce M, et al. Science gateways: the long road to the birth of an institute, Proceedings of the 50th Hawaii International Conference on System Sciences, 2017. Available from:

6. Hsu L, Langseth ML. Community for Data Integration 2017 annual report. U.S. Geological Survey; 2018. doi: 10.3133/ofr20181110

7. Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice, and policy, Health Education Research 1998;13(1):87–108. doi: 10.1093/her/13.1.87 10178339

8. Scheirer MA. Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability, American Journal of Evaluation, 2005. doi: 10.1177/1098214005278752

9. Stirman SW, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research, Implementation Science, 2012. doi: 10.1186/1748-5908-7-17 22417162

10. Stewart CA, Almes GT, and Wheeler BC (eds). Cyberinfrastructure Software Sustainability and Reusability: Report from an NSF-funded workshop. Published by Indiana University, Bloomington, IN. 2010. Available from

11. Wernert J, Wernert EA, and Stewart CA. Models for Sustainability for Robust Cyberinfrastructure Software—Software Sustainability Survey, Indiana University, Bloomington, IN. PTI Technical Report PTI-TR13-007. 2013. Available from:

12. Arp LG, Forbes M, Cartolano RT, Cramer T, Kimpton M, Skinner K, et al. It Takes a Village: Open Source Software Sustainability. Columbia University Academic Commons, 2018. doi: 10.7916/D89G70BS

13. Paskin N. Digital Object Identifiers for scientific data. Data Science Journal. 2006; 12–20. doi: 10.2481/dsj.4.12

14. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific data, 3. 2016. doi: 10.1038/sdata.2016.18 26978244

15. Hsu L, Hutchison VB, and Langseth ML, Data on the Deliverables, Sustainability, and Collaboration of Community for Data Integration Projects from 2010–2016: U.S. Geological Survey, 2019. doi: doi:10.5066/P9V3XDY6

16. Holl S. U.S. Geological Survey Community for Data Integration—NWIS Web Services Snapshot Tool for ArcGIS: U.S. Geological Survey Fact Sheet 2011–3141, 2011. doi: 10.3133/fs20113141

17. Ignizio DA, O’Donnell MS, Talbert CB. Metadata wizard—An easy-to-use tool for creating FGDC–CSDGM metadata for geospatial datasets in ESRI ArcDesktop: U.S. Geological Survey Open-File Report, 2014–1132. 2014. doi: 10.3133/ofr20141132

18. Rosemartin A, Langseth ML, Crimmins TM, Weltzin JF. Development and release of phenological data products—A case study in compliance with federal open data policy: U.S. Geological Survey Open-File Report 2018–1007. 2018. doi: 10.3133/ofr20181007

19. Guy M, Earle P, Horvath S, Turner J, Bausch D, Smoczyk G. Social Media Based Earthquake Detection and Characterization, 2014 KDD Workshop on Learning about Emergencies from Social Information (KDD-LESI 2014), New York City, USA, August 24, 2014. Available from:

20. U.S. Geological Survey. Public Access to Results of Federally Funded Research at the U.S. Geological Survey: Scholarly Publications and Digital Data, 2016. Available from:

21. Henkel HS, Hutchison VB, Chang MY, Zolly L, Uribe R, Faust T. Data Management Website, ScienceBase, 2012. Available from:

22. Faundeen JL, Burley TE, Carlino JA, Govoni DL, Henkel HS, Holl SL, et al. United States Geological Survey Science Data Lifecycle Model: U.S. Geological Survey Open-File Report 2013–1265. 2013, doi: 10.3133/ofr20131265

23. Fundamental Science Practices Advisory Committee. U.S. Geological Survey Fundamental Science Practices: U.S. Geological Survey Circular 1367, 2011.

24. Faundeen JL, Everette AL, Zolly L, Cross VA, Kemp SK, Galvan S, et al. Developing a USGS Legacy Data Inventory to Preserve and Release Historical USGS Data. ScienceBase, 2016. Available from:

25. Chase KJ, Bock AR, Sando R. Sharing our data—An overview of current (2016) USGS policies and practices for publishing data on ScienceBase and an example interactive mapping application: U.S. Geological Survey Open-File Report 2016–1202, 2017, doi: 10.3133/ofr20161202

26. Gordon J, Chkhenkeli N, Govoni D, Lightsom F, Ostroff A, Schweitzer P, et al. A case study of data integration for aquatic resources using semantic web technologies: U.S. Geological Survey Open-File Report 2015–1004, 2015. doi: 10.3133/ofr20151004

27. Plale B, Kouper I. The centrality of data: data lifecycle and data pipelines. In Data Analytics for Intelligent Transportation Systems 2017 Jan 1 (pp. 91–111). Elsevier. doi: 10.1016/B978-0-12-809715-1.00004–3

28. Gundersen LC. Scientific integrity and ethical considerations for the research data life cycle. Scientific Integrity and Ethics: With Applications to the Geosciences. 2017 Oct 17:133–53. doi: 10.1002/9781119067825.ch9

29. Thieler ER, Zeigler SL, Winslow LA, Hines MK, Read JS, Walker JI. Smartphone-based distributed data collection enables rapid assessment of shorebird habitat suitability. PloS one. 2016 Nov 9;11(11):e0164979. doi: 10.1371/journal.pone.0164979 27828974

30. Han T, Rukhlov AS, Riddell JM, Ferbey T. A skeleton data model for geochemical databases at the British Columbia Geological Survey. 2018. Geological Fieldwork 2018, British Columbia Ministry of Energy, Mines and Petroleum Resources, British Columbia Geological Survey Paper 2019–01:125–136.

31. Esri Product Lifecycle Support Policy, 2019 [cited Aug 14, 2019]. Available from

32. Silberschatz A, Gagne G, Galvin PB. Operating system concepts. Wiley; 2018 Jan 18.

33. Earth Resources Observation and Science Center, Adopt a Pixel, 2019, doi: 10.5066/F7VM4B7C

34. Global Learning to Benefit the Environment (GLOBE) Data User Guide, 2019, version 1.0,

35. Global Learning and Observations to Benefit the Environment (GLOBE) Program, 2019 Jul 26,

36. Hayden L, Taylor J, Robles MC. GLOBE: Connecting to Community of Observers Directly to NASA Satellites [Education]. IEEE Geoscience and Remote Sensing Magazine 2019;7(1):98–99. doi: 10.1109/MGRS.2019.2891930

37. DeRisi S, Kennison R, Twyman N. The What and Whys of DOIs, PLoS Biol 2003;1(2):e57. doi: 10.1371/journal.pbio.0000057 14624257

38. Boudry C, Chartron G. Availability of digital object identifiers in publications archived by PubMed, Scientometrics 2017;110:1453–1469. doi: 10.1007/s11192-016-2225-6

39. Hall KL, Vogel AL, Huang GC, Serrano KJ, Rice EL, Tsakraklides SP, Fiore SM. The science of team science: A review of the empirical evidence and research gaps on collaboration in science. American Psychologist. 2018 May;73(4):532. doi: 10.1037/amp0000319 29792466

40. Goring SJ, Weathers KC, Dodds WK, Soranno PA, Sweet LC, Cheruvelil KS, et al. Improving the culture of interdisciplinary collaboration in ecology by expanding measures of success. Frontiers in Ecology and the Environment 2014;12:39–47, doi: 10.1890/120370

41. Stokols D, Hall KL, Taylor BK, Moser RP. The science of team science: overview of the field and introduction to the supplement. American journal of preventive medicine. 2008 Aug 1;35(2):S77–89. doi: 10.1016/j.amepre.2008.05.007

42. Stokols D, Misra S, Moser RP, Hall KL, Taylor BK. The ecology of team science: understanding contextual influences on transdisciplinary collaboration. American journal of preventive medicine. 2008 Aug 1;35(2):S96–115. doi: 10.1016/j.amepre.2008.05.003 18619410

43. Hall KL, Vogel AL, Stipelman BA, Stokols D, Morgan G, Gehlert S. A four-phase model of transdisciplinary team-based research: goals, team processes, and strategies. Translational behavioral medicine. 2012 Oct 25;2(4):415–30. doi: 10.1007/s13142-012-0167-y 23483588

44. Office of Management and Budget (OMB), M-05-03 Final Information Quality Bulletin for Peer Review. [cited 2019 Aug 12] Available from:

45. Narock T, Goldstein EB, Jackson CA, Bubeck AA, Enright AML, Farquharson JI, et al. Earth science is ready for preprints, Eos, 2019;100, doi: 10.1029/2019EO121347

46. U.S. Geological Survey Manual 502.4—Fundamental Science Practices: Review, Approval, and Release of Information Products. [cited 2019 Aug 12] Available from:

47. Otte E, Rousseau R. Social network analysis: a powerful strategy, also for the information sciences. Journal of Information Science 2002;28(6):441–453. doi: 10.1177/016555150202800601

48. Zoss AM, Börner K. Mapping interactions within the evolving science of science and innovation policy community. Scientometrics 2011;91(2):631–44. doi: 10.1007/s11192-011-0574-8

49. Hicks DJ, Coil DA, Stahmer CG, Eisen JA. Network analysis to evaluate the impact of research funding on research community consolidation. PLoS ONE 2019;14(6):e0218273. doi: 10.1371/journal.pone.0218273 31211808

Článek vyšel v časopise


2019 Číslo 10