Identifying publications in questionable journals in the context of performance-based research funding

Autoři: Joshua Eykens aff001;  Raf Guns aff001;  A. I. M. Jakaria Rahman aff001;  Tim C. E. Engels aff001
Působiště autorů: Centre for R&D Monitoring (ECOOM), Faculty of Social Sciences, University of Antwerp, Antwerp, Belgium aff001;  Department of Communication and Learning in Science, Chalmers University of Technology, Gothenburg, Sweden aff002
Vyšlo v časopise: PLoS ONE 14(11)
Kategorie: Research Article
doi: 10.1371/journal.pone.0224541


In this article we discuss the five yearly screenings for publications in questionable journals which have been carried out in the context of the performance-based research funding model in Flanders, Belgium. The Flemish funding model expanded from 2010 onwards, with a comprehensive bibliographic database for research output in the social sciences and humanities. Along with an overview of the procedures followed during the screenings for articles in questionable journals submitted for inclusion in this database, we present a bibliographic analysis of the publications identified. First, we show how the yearly number of publications in questionable journals has evolved over the period 2003–2016. Second, we present a disciplinary classification of the identified journals. In the third part of the results section, three authorship characteristics are discussed: multi-authorship, the seniority–or experience level–of authors in general and of the first author in particular, and the relation of the disciplinary scope of the journal (cognitive classification) with the departmental affiliation of the authors (organizational classification). Our results regarding yearly rates of publications in questionable journals indicate that awareness of the risks of questionable journals does not lead to a turn away from open access in general. The number of publications in open access journals rises every year, while the number of publications in questionable journals decreases from 2012 onwards. We find further that both early career and more senior researchers publish in questionable journals. We show that the average proportion of senior authors contributing to publications in questionable journals is somewhat higher than that for publications in open access journals. In addition, this paper yields insight into the extent to which publications in questionable journals pose a threat to the public and political legitimacy of a performance-based research funding system of a western European region. We include concrete suggestions for those tasked with maintaining bibliographic databases and screening for publications in questionable journals.

Klíčová slova:

Careers – Citation analysis – Health economics – Medicine and health sciences – Open access publishing – Peer review – Research funding – Scientific publishing


1. Hern A, Duncan P. Predatory publishers: the journals that churn out fake science. The Guardian. 10 Aug 2018. Available from:

2. Debusschere B, Amkreutz R. Opzettelijke fraude of in de val gelopen? Alle Vlaamse universiteiten te vinden in rooftijdschriften. De Morgen. 10 Aug 2018. Available from: (Dutch)

3. Alecci, S. “New international investigation tackles ‘fake science’ and its poisonous effects”. Inernational Consortium of Investigative Journalists. 20 July 2018. Available from:

4. Beall J. (2010). ‘Predatory’ open-access scholary publishers. The Charleston Advisor, 11, 10–17.

5. Bornmann L. Scientific Peer Review: An Analysis of the Peer Review Process from the Perspective of Sociology of Science Theories. Human Architecture: Journal of the Sociology of Self-Knowledge. 2008; 6(2): 23–38.

6. Hicks D. Overview of Models of Performance-Based Research Funding Systems., in: OECD, Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings. OECD Publishing, Paris, 2010, pp. 23–52.

7. Engels T. C. E., Guns R. The Flemish performance-based research funding system: A unique variant of the Norwegian model. Journal of Data and Information Science. 2018; 3(4): 44–59.

8. Memon A. R. Revisting the Term Predatory Open Access Publishing. Journal of Korean Medical Science. 2019; 34(13): 1–7.

9. Aagaard K. How incentives trickle down: Local use of a national bibliometric indicator system. Science and Public Policy. 2015; 42(5): 725–737.

10. Sīle L, Guns R, Sivertsen G, Engels TCE. European Databases and Repositories for Social Sciences and Humanities Research Output. ECOOM & ENRESSH, Antwerp, 2017. Available from:

11. Sīle L, Pölönen J, Sivertsen G, Guns R, Engels TCE, Arefiev P, et al. Comprehensiveness of national bibliographic databases for social sciences and humanities: findings from a European survey. Research Evaluation. 2018; 27(4): 310–322.

12. Verleysen F, Ghesquière P, Engels TCE. The objectives, design and selection process of the Flemish Academic Bibliographic Database for the Social Sciences and Humanities (VABB-SHW), in: Blockmans W., Engwall L., Weaire D. (Eds.), Bibliometrics: Use and Abuse in the Review of Research Performance, Portland Press, London, 2014: pp. 117–127.

13. Eriksson S, Gelgesson G. Time to stop talking about ‘predatory journals’. Learned publishing. 2018; 31(2): 181–183.

14. Eykens J, Guns R, Rahman A.I.MJ, Sīle L, Engels TCE. Predatory Open Access Journals: A review of past screenings within the Flemish performance based research funding system 2014–2018). STI 2018 Conference Proceedings. 2018: pp. 799–806. Available from:

15. Guns R, Sīle L, Eykens J, Verleysen FT, Engels TCE. A comparison of cognitive and organizational classification of publications in the social sciences and humanities. Scientometrics. 2018; 116(2): 1093–1111.

16. Rahman AIMJ, Dexters N, Engels TCE. Predatory open access journals in a performance-based funding model: Common journals in Beall’s list and in the VABB-SHW. ECOOM, Antwerp, 2014. Available from:

17. Rahman AIMJ, Engels TCE. Predatory open access journals in a performance-based funding model: Common journals in Beall’s list and in version V of the VABB-SHW. ECOOM, Antwerp, 2015. Available from:

18. Rahman AIMJ, Guns R, Engels TCE. Predatory open access journals in a performance-based funding model: A comparison of journals in version VI of the VABB-SHW with Beall’s list and DOAJ. ECOOM, Antwerp, 2015. Available from:

19. Sīle L, Guns R, Engels TCE. Comparing VABB-SHW (version VII) with Beall’s lists and DOAJ. ECOOM, Antwerp, 2017. Available from:

20. Eykens J, Guns R, Engels TCE. Comparing VABB-SHW (version VIII) with Cabells Journal Blacklist and Directory of Open Access Journals: report tot he Authoritative Panel. ECOOM, Antwerp, 2018. Available from:

21. Butler D. Investigating journals: The dark side of publishing. Nature. 2013; 495(7442): 433–435. doi: 10.1038/495433a 23538810

22. ABDC. ABDC Journal Quality List Initial Interim Review Outcomes. Deakin, Australia. Australian Business Deans Council (ABDC) Inc.; 2016.

23. Beall J. Predatory Publishing is just one of the consequences of gold open access. Learned Publishing. 2013; 26: 79–84.

24. Beall J. Predatory Publishing: Overzealous open-access advocates are creating an exploitative environment, threatening the credibility of scholarly publishing, The Scientist. 2012. Available from:

25. Bloudoff-indelicato M. Backlash after Frontiers journals added to list of questionable publishers. Nature. 2015; 526(7575): 613.

26. Olijhoek T. Quality of DOAJ Listed Journals, DOAJ New Service. (2019). Available from:

27. Strinzel M, Severin A, Milzow K, Egger M. “Blacklists” and “whitelists” to tackle predatory publishing: A cross-sectional comparison and thematic analysis. PeerJ Preprints. 2019; 7:e27532v1.

28. Silver AJ. Controversial website that lists ‘predatory’ publishers shuts down. Nature. 2017.

29. Xia J, Harmon JL, Connolly KG, Donnelly RM, Anderson MR, Howard HA. Who publishes in “predatory” journals?. Journal of the Association for Information Science and Technology. 2015; 66(7): 1406–1417.

30. Frandsen TF. Are predatory publishers undermining the credibility of science? A bibliometric analysis of citers. Scientometrics. 2017; 113(3): 1513–1528.

31. Shen C, Björk BC. ‘Predatory’ open access: a longitudinal study of article volumes and market characteristics. BMC Medicine. 2015; 13(320).

32. Moher D, Shamseer L, Cobey KD, Lalu MM, Galipeau J, Avey MT, et al. Stop this waste of people, animals and money. Nature. 2017; 549: 23–25. doi: 10.1038/549023a 28880300

33. Olijhoek T, Tennant J. The “problem” of predatory publishing remains a relatively small one and should not be allowed to defame open access, LSE Impact Blog. (2018).

34. Kurt S. Why do authors publish in predatory journals?. Learned Publishing. 2018; 31(2): 141–147.

35. Wallace F. H., Perri T. J. Economists behaving badly: publications in predatory journals. 2018; 115(2): 749–766.

36. Nicholas D, Jamali HR, Watkinson A, Herman E, Tenopir C, Volentine R, et al. Do Younger Researchers Assess Trustworthiness Differently when Deciding what to Read and Cite and where to Publish?. International Journal of Knowledge Content Development and Technology. 2015; 5(2): 45–63.

37. Leišytė L. New public management and research productivity–a precarious state of affairs of academic work in the Netherlands. Studies in Higher Education. 2016; 41(5): 828–846.

38. Eykens J, Guns R. Identifying publications in questionable journals in the context of PRFS: code and data (v 1.0). Zenodo. 2019.

39. OECD. Revised Field of Science and Technology (FOS) Classification in the Frascati Manual. Working Party of National Experts on Science and Technology. 2007. Available from:

40. Verleysen F. T., Weeren A. Clustering by publication patters of senior authors in the social sciences and humanities. Journal of Informetrics. 2016; 10(1): 254–272.

41. Bearlocher MO, Newton M, Gautam T, Tomlinson G, Detsky AS. The meaning of Author Order in Medical Research. Journal of Investigative Medicine. 2015; 55(4): 174–180.

42. Moher D, Srivastava A. Your are invited to submit…. BMC Medicine. 2015; 13(1): 180–184.

43. Manca A, Martinez G, Cugusi L, Dragone D, Dvir Z, Deriu F. The surge of predatory open-access in neurosciences and neurology. Neuroscience. 2017; 353: 166–173. doi: 10.1016/j.neuroscience.2017.04.014 28433651

44. Guns R. Alphabetical co-authorship in the social sciences and humanities: evidence from a comprehensive local database, in: I. Rafols (Eds.), Proceedings of the 21st International Conference on Science and Technology Indicators, Valencia, 2016: pp. 957–964. Available from:

45. Engels TCE, Ossenblok TLB, Spruyt EHJ. Changing publication patterns in the Social Sciences and Humanities, 2000–2009. Scientometrics. 2012; 93(2): 373–390.

46. Deprez E. E., Chen C. Medical Journals Have a Fake News Problem. Bloomberg Businessweek. 29 Aug 2017. Available from:

47. Bartholomew R. E. Science for sale: the rise of predatory journals. Journal of the Royal Society of Medicine. 2014; 107(10): 384–385. doi: 10.1177/0141076814548526 25271271

48. Demir S. B. Predatory journals: Who publishes in them and why?. Journal of Informetrics. 2018; 12(4): 1296–1311.

49. Shaghaei N., Wien C., Holck J. P., Thiesen A. L., Ellegaard O., Vlachos E., et al. Being a deliberate prey of a predator: Researchers’ thoughts after having published in predatory journal. LIBER Quarterly. 2018; 28: 1–17.

Článek vyšel v časopise


2019 Číslo 11