An Evaluation of Primary Studies Published in Predatory Journals Included in Systematic Reviews From High-Impact Dermatology Journals: Cross-sectional Study

Introduction

Predatory publishing is described as a “nebulous concept of research journal publishers who use unethical business practices, minimal or no peer review, or limited editorial oversight to publish articles that are below a minimally accepted standard of quality” [1]. Increasing rates of predatory publishing are accompanied by an equally growing concern surrounding their threat to evidence synthesis and decision-making [1,2]. Predatory publishing can be problematic to our highest standard of scientific evidence—systematic reviews (SRs)—through the inclusion of poor-quality and unusable data, which could mislead results, challenge outcomes, and undermine confidence due to suspected predatory journals (SPJs) having a less rigorous peer review process.[3] Evidence is lacking as to whether studies published in SPJs are frequently included as primary studies in SRs; therefore, we aimed to evaluate whether SRs published in top dermatology journals contain primary studies published in SPJ.

Methods

We searched PubMed (using the Advanced Search filters) for SRs published in the top five dermatology journals (determined by 5-year h-indices) between January 01, 2019, and May 24, 2021. The returned SRs (N=339) were downloaded as a comma-separated values file. We randomized the returns and selected the first 100 articles to examine. Primary studies were extracted from each systematic review, and the publishing journal of these primary studies was cross-referenced using Beall’s List (archived and updated versions [4]) and the Directory of Open Access Journals (DOAJ) [5], both widely used and publicly available databases of suspected predatory or questionable journals. To determine if certain study characteristics were associated with the inclusion of SPJs, the following characteristics were extracted: (1) whether the SR received funding; (2) whether the SR had a registered protocol; (3) whether the SR included randomized controlled trials, nonrandomized studies of interventions, or both as primary studies; (4) the year the SR was published; and (5) the databases the SR searched for primary studies, to determine if certain study characteristics were associated with the inclusion of SPJs. Screening and data extraction were performed in a masked, duplicate fashion by authors BH and KS, in accordance with best practices [6]. We performed chi-square tests to determine possible associations between an SR’s inclusion of a primary study published in an SPJ and particular study characteristics.

This study did not use human subjects and thus did not require institutional review board oversight.

Results

Our randomized sample included 100 SRs, of which 31 (31%) SRs were found to contain a primary study published in an SPJ. A total of 53 primary studies were published across 22 unique SPJs. Of the top five dermatology journals, the Journal of the American Academy of Dermatology had the most SRs containing a primary study published in an SPJ (Table 1). The majority of suspected predatory publications (28/55, 51%) were published in the Indian Journal of Dermatology, Venereology, and Leprology. SRs that contained a meta-analysis were significantly less likely to contain a primary study published in an SPJ (P=.002; Table 1). Additionally, SRs that had a registered protocol were less likely to contain a primary study published in an SPJ (P=.02). No statistically significant associations were found between journals, year of publication, included primary study types (eg, randomized controlled trials, nonrandomized studies of interventions, or both), funding, or databases included in the SR search.

Discussion

We found that studies published in SPJs are commonly included as primary studies in SRs published in high-impact dermatology journals. SRs that contained a meta-analysis were less likely to have a primary study published in an SPJ, which is a promising finding, as research has shown that studies published in predatory journals are of lower quality [1,3]. Interestingly, SRs that registered a protocol were significantly less likely to include a primary study published in an SPJ. We suspect this finding may be because authors of SRs with registered protocols may have more diligence and time to confirm that sources of publications were not published in an SJP. In our sample, the majority of primary studies from SPJs were published in the Indian Journal of Dermatology, Venereology, and Leprology—which was removed from the DOAJ directory secondary to the journal failing to adhere to best practice [5]. Although considered to be an SPJ, this journal’s articles are included in Embase and PubMed searches. Interestingly, 83% (44/53) of the studies published in SPJs were PubMed indexed. One way through which studies published in SPJs can obtain PubMed indexing is “backdoor publishing” via PubMed Central or the National Center for Biotechnology Information Bookshelf [7]. Currently, there is little direction on how to best manage SPJs; however, the consensus is that studies published in SPJs should be omitted because of their potential impact on data synthesis. Due to their potential threat to SRs and scientific evidence, we recommend that authors of SRs verify their primary studies by using Beall’s List and the DOAJ directory—a recommendation proposed by other studies exploring ways to minimize the inclusion of studies published in SPJs in SRs [8,9].

Our study’s limitations include only searching SRs using PubMed and only using Beall’s List and DOAJ lists of questionable journals. Additionally, authors of SRs included in this study may have unknowingly included an SPJ, as some SPJs were added to Beall’s List and the DOAJ lists of questionable journals after the SR was already published, which is another limitation of our study. Lastly, future research is needed to investigate the effects of including SPJ publications in scientific research.