Our final analysis of 280 dermatology publications included 127 publications (45.4%) with empirical data from reproducible study designs and 153 publications (54.6%) that lacked empirical data or were inherently difficult to reproduce. The median 5-year journal impact factor was 2.719. Journal impact factors were inaccessible for 21 publications. Tables 2 and 3 provide additional characteristics for our sample of dermatology publications.

Among the 280 eligible publications, 201 (71.8%) were publicly available, whereas the remaining 79 (28.2%) were only available through a paywall. We classified the 20 publications for which full text PDF versions were unattainable as being paywall restricted. Thus, a total of 99 publications (of 300; 33.0%) were classified as being unavailable to the public. Only 23 publications (out of 114, 20.2%) provided a statement indicating that additional materials were available. Only 3 publications (out of 127, 2.4%) provided a protocol availability statement. All 3 of these statements provided a valid link to a Web-based protocol. Almost all publications lacked data availability statements. A total of 14 publications (out of 127, 11.0%) included data availability statements; however, only 11 of these data statements were linked to supplemental data files. Of the 11 accessible supplemental data files, only 3 provided access to complete and unmodified raw datasets. In addition, only 1 publication (out of 127, 0.8%) provided an analysis script or code. Our analysis revealed only 3 publications (of 127, 2.4%) were prospectively registered. A total of 233 publications (out of 280, 83.2%) provided a COI statement. Of these 280 publications, 30 (10.7%) indicated that 1 or more authors had a COI, and 203 (72.5%) declared the author(s) did not have a COI. The remaining 47 publications (out of 280, 16.8%) failed to provide a COI statement. Furthermore, 155 (out of 280, 55.4%) publications reported a funding source, whereas 125 (44.6%) publications did not receive external funding. Finally, 23 publications (out of 114, 20.2%) included in our analysis were cited in a subsequent data synthesis or review paper (Table 4). No publication included in our analysis was cited in a replication study.

Our findings suggest that the current climate of dermatology research does not encourage reproducible and transparent research practices. Few studies provided access to datasets, analysis scripts, or complete study protocols. These findings are congruent with previous reports that found that studies often fail to promote transparent and reproducible research practices [9], and they align with a study published in Nature that found that 90% of more than 1500 researchers agreed that biomedical science is facing a significant reproducibility crisis [1]. This environment of poor research practice is problematic for clinicians and researchers who might seek to validate or reproduce a study in its entirety. As scientists and clinicians continue to make medical advances, studies must be readily reproducible to ensure proper validation of results and to allow for sustained progression in clinical practice. In the following text, we describe 2 practices in the field of dermatology—study protocols and preregistration—that were commonly omitted by researchers. We follow with actionable recommendations for research funders, journals, and researchers that, if implemented successfully, might help better the climate of reproducible research in published dermatology literature.

Most studies included in our sample did not provide additional materials or complete study protocols. Precisely outlining methodology is essential for study reproducibility [31], whether this information is provided within the publication or in supplementary materials [32]. The Journal of the American Academy of Dermatology’s (JAAD) instructions to authors state, “submissions of research articles should be accompanied by a supplementary document that includes the protocol and statistical analysis plan; this should be labeled ‘For editor/reviewer reference only’ and is not for publication” (emphasis ours) [33]. The British Journal of Dermatology (BJD) author guidelines state, “The editorial team has found that providing the study protocol facilitates acceptance of the paper if it is available. Therefore, the BJD encourages submission of the protocol at the time of manuscript submission, with the protocol identified as a ‘Supplementary file for review.’ Submission of the trial protocol is also strongly encouraged for industry-sponsored trials.” [34] JAMA Dermatology guidance states, “authors of manuscripts reporting clinical trials must submit trial protocols (including the complete statistical analysis plan) along with their manuscripts… and that if the manuscript is accepted, the protocol and statistical analysis plan will be published as a supplement [35].” The widespread variability in guidance provided by these 3 prominent dermatology journals—which ranges from nonpublication of study protocols by JAAD to protocol publication upon article acceptance by JAMA Dermatology—suggests differing views toward implementing reproducible research practices within the field. BJD does not require protocol submission but simply encourages it. As journals are the final arbiters of studies that move on to publication, they have a high degree of influence on the climate of reproducibility and transparency in dermatology research. We highly recommend that dermatology journals adopt stronger requirements for submitting authors to promote greater transparency and reproducibility.

According to the Food and Drug Administration Amendments Act, established in 2007, all applicable RCTs must be registered before participant enrollment [22]. Although the number of preregistered RCTs has increased, other study designs have not shown as much improvement. Boccia et al found that only 1109 cancer observational studies were registered on ClinicalTrials.gov across an 11-year period [36]. In addition, systematic reviews have a preregistration platform, the International Prospective Register of Systematic Reviews (PROSPERO), which has increased in usage exponentially since its inception in 2011 [37]. These study designs are preregistered solely at the authors’ discretion, with few journals or funders having concrete guidance on the subject. Of the 3 journals discussed above, only BJD mentions registering systematic reviews, stating that authors are required to preregister on PROSPERO [34]. Transparent research practices such as prospective registration can help mitigate unethical research practices by providing access to date-stamped protocol details and informing the public about current clinical trials being performed [38]. For example, P-hacking (using different statistical analyses until a nonsignificant finding is found to be significant) [39] and HARKing (forming study hypothesis after results have been calculated) [40] might be avoided if investigators disclose the expected statistical analyses that will be used throughout the study before its commencement. It should be noted that HARKing can be beneficial to the scientific process by generating important discoveries during post hoc analyses [41-43] In addition, previous studies have shown that reviewers often encourage authors to add hypotheses post hoc as part of the peer review process [44]. However, the crossover into research misconduct occurs when authors contend that these posthoc hypotheses were part of the original study design, thereby potentially decreasing the confidence of statistically significant outcomes [45].

Changes to the landscape of dermatology research are warranted; however, the optimal framework for doing so is unclear. Here, we offer recommendations for research stakeholders—including funding agencies, journals, and researchers—that may help increase the quality of reproducible research practices in dermatology, if implemented successfully.

With respect to funding, some foundations and governmental agencies have established measures to promote reproducibility and transparency of research for which they provide funding. A nonexhaustive list of these funders include the National Institutes of Health (NIH), the National Science Foundation, the Wellcome Trust, and the Bill and Melinda Gates Foundation. As one example, the Gates Foundation, which funds approximately 2000 to 2500 research articles per year totaling US $5 billion [46], has established an open access policy requiring that all research data and manuscripts resulting from its funds be promptly and broadly disseminated [47]. To further its goals for widespread dissemination, the foundation has launched its own open access journal, Gates Open Research. Currently, research funded by the foundation is not eligible for publication in some of the world’s most renowned journals, such as Nature, Science, Proceedings of the National Academy of Sciences, and New England Journal of Medicine owing to these funding restrictions [48]. The NIH has established the Rigor and Reproducibility Initiative, embedding requirements that submitted grant applications outline strategies for more reproducible research [49]. Strategies such as these are the first steps toward adoption of more transparent and reproducible research practices.

For journals, we recommend consideration of adopting stricter standards on the disclosure of study materials, raw datasets, protocols, and analysis scripts. Journals should consider requiring that authors share all study materials on public repositories, such as Open Science Framework. With essential study materials publicly available, outcomes may be reproduced and validated with greater ease. A recent survey found that open access to study data increased the public’s trust and confidence in research outcomes [50]. Depositing all study materials and data before publication may increase the public’s faith and confidence in the literature published in journals with such requirements.

Finally, for researchers, we believe a need exists to train and equip principal investigators to adopt more reproducible and transparent research practices. This goal may be best accomplished through continuing education, academic conferences, webinars, and journal clubs. A need also exists to train and equip the next generation of scientists. Given the apprenticeship nature of many biomedical laboratories, principal investigators should take the lead in fostering such cultures within their laboratories and instilling such practices with mentees. Courses on open science are being developed across the country, many posted on the Open Science Framework [51]. The National Institutes of General Medical Sciences has posted several Web-based training modules to increase the overall rigor and reproducibility of medical research [52]. As these courses continue to expand at universities and with funders, continued development and uptake of such training may help reverse the scant nature of reproducibility and transparency of research in the dermatology literature.

Our study has many strengths, but some limitations are present. Regarding strengths, all materials, protocols, analysis plans, and raw data from our study are publicly available on Open Science Framework. In addition, we implemented numerous measures to ensure the reliability of study outcomes by (1) using a blinded, double data extraction technique—the gold standard for meta-research practices [53] and (2) providing thorough training of each investigator to ensure reliability of results between investigators. Regarding limitations, data extraction was limited to the content of the full-text PDFs and available supplemental materials for each publication. Additional materials may be attainable by contacting the corresponding author. Furthermore, this study focused specifically on publications in dermatology journals. Thus, the results from this study may not be generalizable to other subjects or years of publication. For the aforementioned reasons, interpretation of our findings should be considered a lower bound estimate of reproducibility of publications in dermatology journals.

In conclusion, the rate of disclosure of study materials, data, protocols, and analysis scripts of sampled dermatology publications is unacceptably low. Without implementing and adhering to more robust reporting standards and open science practices, reproducibility-related factors of dermatologic research may remain poor.