To enhance the transparency and reproducibility of our work, we have provided our study materials, methods, and protocol on Open Science Framework [24]. While drafting this manuscript, we followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [25] and Murad and Wang’s guidelines for reporting meta-epidemiological methodology [26].

We searched MEDLINE (Ovid) and Embase (Ovid) on June 2, 2020, for systematic reviews with or without meta-analyses specific to the treatment of melanoma. Our exact search string can be found in Multimedia Appendix 1.

Two authors (ZR and KP) screened search returns by title and abstract in a duplicate, masked manner. After title and abstract screening, full texts were screened according to the eligibility criteria described below. Discrepancies were resolved by a group consensus meeting, with third-party adjudication, if necessary.

To be included, articles must have (1) been considered a systematic review or meta-analysis according to the PRISMA-P (PRISMA for Protocols) definition [27]; (2) been a head-to-head comparison of a specific intervention, or combination of interventions, to another intervention or to a placebo or standard of care; (3) been specific to the treatment of melanoma; (4) been published between the dates of September 1, 2016, and June 2, 2020; (5) been published in English; and (6) synthesized data from human participants. The dates of inclusion were based on the International Committee of Medical Journal Editors (ICMJE) recommendation, which states that authors should disclose COIs that occurred 36 months prior to journal submission [28]. By including systematic reviews published after September 1, 2016, we were able to cross-reference reported payments on the CMS Open Payments database—which went live in September 2013—in the 36 months prior to the dates of publication of the systematic reviews within our sample to ensure compliance with ICMJE’s COI disclosure policy.

Before the study began, investigators received an online training overview. Training included details regarding the study design, objectives, protocol, materials, and data extraction from one systematic review as an example. This training session is available online for reference [24].

The same investigators (ZR and KP) who performed study screening also completed data extraction in a masked, duplicate fashion using a pilot-tested Google Form. The full texts of the included studies were analyzed for general study characteristics, including the following: (1) PubMed identification number and/or DOI (Digital Object Identifier), (2) name of journal, (3) date of publication, (4) author names, (5) treatment interventions being compared, (6) affiliations for the first and last authors, (7) funding source, (8) complete COI statement, (9) whether the systematic review or meta-analysis addressed risk of bias (RoB), (10) the verbatim RoB statement, (11) whether a systematic review author was also an author on one or more of the primary studies included in the review (yes or no), (12) the total number of self-cited primary studies, (13) whether an overall pooled effect estimate was calculated (yes or no), (14) the statistical significance of the pooled effect estimate, and (15) whether narrative results and conclusions favored the treatment or comparison group (eg, placebo, standard of care, or control). For the purpose of our study, conclusion was defined as the combined discussion and conclusion sections of the review.

Narrative results and conclusions were deemed as favorable, unfavorable, or mixed or inconclusive. While appraising the results section, favorable was assigned when only positive results were reported. Unfavorable was assigned when only negative results were reported. Mixed or inconclusive was assigned if both positive and negative results were reported. While appraising the conclusion, favorable was assigned when authors stated or implied favorability of the intervention group over the comparator group. Unfavorable was assigned when authors stated or implied favorability of the comparator group over the intervention group. Mixed or inconclusive was assigned if the conclusion section did not meet criteria for favorable or unfavorable (eg, reporting negative population outcome but positive subgroup analysis).

Our search for undisclosed COIs was undertaken using the stepwise strategy outlined in Figure 1. We used a similar search strategy used by Mandrioli et al [29], with slight modifications. These modifications included the use of additional databases: the CMS Open Payments database, Dollars for Profs, and the United States Patent and Trademark Office (USPTO). Multimedia Appendix 2 describes each database. All authors were searched for within these databases for undisclosed COIs, regardless of disclosure status. To ensure the accuracy of data collection, MW used the Python programming language (Python Software Foundation) to create database-specific search strings for the USPTO, Google Patents, and PubMed. If results from the patent searches could not be definitively linked to the author in question, we erred on the side of caution and did not consider this as an undisclosed COI. Based on recommendations for COI disclosure offered by the ICMJE, we limited our search of PubMed to include studies published in the 36 months prior to the date of the systematic review included in our sample. Author COI disclosure statements from the PubMed search results were cross-referenced with the COI disclosure statement found in the systematic review from our sample to determine if previously published studies included additional COIs not disclosed in the systematic review from our sample. In the event that more than 10 records were returned from our PubMed search, each investigator (ZR and KP) individually assigned random numbers to the records and screened the first 10 randomized records for an undisclosed COI. This stepwise search process was continued until an undisclosed COI was discovered, at which point the search was terminated and the author was considered to have an undisclosed COI [29].

We used the Cochrane Collaboration’s criteria to assess the risk of funding bias in the included systematic reviews, as well as the four items used by Mandrioli et al [29]. Overall RoB was determined using the following criteria: (1) selection for inclusions and exclusions was explicit and well-defined and could be replicated by others, (2) the study inclusion method involved two or more assessors selecting studies, (3) search strategies were comprehensive, and (4) studies controlled for methodological differences that may introduce bias. We considered the RoB to be high if fewer than three items received a satisfactory yes answer.

Results were calculated and reported as descriptive statistics. Relationships between systematic review characteristics and outcomes were evaluated by Fisher exact tests, when possible. Stata 16.1 (StataCorp, LLC) was used for all analyses.

Our search of MEDLINE and Embase yielded 2388 records. A total of 2312 records were excluded after title and abstract screening. Full-text screening led to the exclusion of an additional 53 records. A total of 23 systematic reviews with or without meta-analyses investigating treatment interventions for melanoma were included (Figure 2).

The 23 systematic reviews included in our final sample were conducted by 120 authors and published within 21 journals. Systematic reviews investigated pharmacologic interventions (8/23, 35%), surgical interventions (7/23, 30%), or a multidisciplinary treatment approach (8/23, 35%). Of the 23 systematic reviews, 19 (83%) reported that none of the authors had a COI, 2 (9%) reported that at least one author had a COI, and 2 (9%) failed to provide a COI disclosure statement. Only 1 systematic review out of 23 (4%) was found to have a high RoB. Additional study characteristics are provided in Table 1.

Of the 120 authors, 25 (20.8%) were found to have a COI, either disclosed, undisclosed, or both. Of these 25 authors, 20 (80%) reported no COI within the review’s disclosure statement but were found to have an undisclosed COI. The remaining 5 authors out of 25 (20%) disclosed one or more COI but were found to have an additional undisclosed COI that was omitted from the COI disclosure statement (Table 1).

Of the 12 systematic reviews with one or more authors with a COI, 7 (58%) reported narrative results favoring the treatment group and 9 (75%) reported conclusions favoring the treatment group. Of the 11 systematic reviews with no conflicted authors, 4 (36%) reported results favoring the treatment group and 5 (46%) reported conclusions favoring the treatment group. Our results showed no statistically significant association between author COIs and the favorability of results (P=.53) or conclusions (P=.15) (Table 2).

Of the 23 systematic reviews, 9 (39%) received funding support, 8 (35%) did not receive funding support, and 6 (26%) did not provide a funding statement. Of the 9 reviews receiving nonindustry support, 3 (33%) reported results favoring the treatment group and 3 (33%) reported conclusions favoring the treatment group (Table 3). Because our sample did not include a single industry-funded systematic review, we could not assess for a relationship between industry sponsorship and the favorability of review results and conclusions.

Only 1 of the 23 systematic reviews (4%) was found to have a high RoB. This systematic review received nonindustry support and was conducted by one or more authors with at least one COI. Because only 1 systematic review had a high RoB, we were unable to determine whether high RoB influenced the nature of review results and conclusions.

The results of our study indicate that COIs are a regular, often incompletely disclosed, occurrence in systematic reviews investigating melanoma interventions. Roughly one-half of the included systematic reviews were authored by at least one author with a potential COI. Additionally, one-fifth of the systematic review authors did not fully disclose all potential COIs. Previous work using the CMS Open Payments database to detail physician-industry relationships found variable rates of undisclosed COIs among clinical practice guidelines authors in multiple disciplines [30-33]. For example, undisclosed COIs were found to be present for 45% (22/49) of authors in dermatology [30], 6% (3/49) of authors in otolaryngology [31], 31% (23/74) of authors in orthopedics [32], and 20% (20/54) of authors in urology [33]. Similarly, of the 9 authors from our sample who were found on the CMS Open Payments database, all had at least one undisclosed COI that was omitted from the systematic review COI disclosure statement. In an assessment of the association between COIs and results, conclusions, and methodological quality, Hansen et al [34] found that systematic reviews with a COI were more likely to have favorable conclusions than those without a COI. Although our analysis failed to identify a similar association between author COIs and review outcomes, the high rates of undisclosed COIs among authors included in our sample highlight the need for more complete COI disclosure. Inconsistency in the completeness of COI disclosure is evident, and one potential explanation may be a lack of adherence to a comprehensive, more uniform disclosure guideline.

Complete disclosure of COIs is a widespread issue, and the lack of standardization of disclosure requirements between journals could partially explain the high rate of undisclosed COIs in our sample. For example, a study by Zhu and Sun [35] found that only 31% of medical journals mentioned a COI policy, 7% required a COI statement, and 4% standardized the COI submissions form. In addition to the inconsistent presence of journal COI disclosure policies, journals often fail to clearly outline expectations regarding COI disclosure requirements, making it difficult or impossible to establish what COI information should be disclosed. For instance, a 2007 study by Ancker and Flanagin [36] determined that only 68% of journals provided examples of what may be perceived as a potential COI and only 46% of journals explicitly defined the term. The same authors reported, upon initial attempts, that they were only able to locate COI disclosure policies for 33% of “high-impact, peer-reviewed” journals. Results from studies such as these highlight that, even when COI disclosure policies are present, authors are often left to determine for themselves what information should be disclosed at the time of manuscript submission.

Resnik and Elliott [37] reached a similar conclusion concerning the potential influence of financial biases on the design and interpretation of the study. These authors highlighted the difficulties in judging a study on scientific merit alone and presented methods to take financial relationships into account, without crudely discrediting the results of the study. To help address financial bias in medical literature, we recommend an initiative to develop a similar database to the CMS Open Payments database that would centralize potential COIs, financial and otherwise, and include all stakeholders in academic medicine (eg, clinicians, researchers, editors, funders, and peer reviewers), as well as research stakeholders with non-US–based affiliations.

Furthermore, we encourage readers to consider COIs when interpreting the results of systematic reviews. Analyses designed to define the prevalence of undisclosed and disclosed COIs in medical literature may increase awareness and emphasis on the issues surrounding COIs and lead to more standardized disclosure policies, such as an ICMJE COI form expanded to a global scale [20]. Perhaps a more comprehensive and enforced implementation of a standardized COI form would decrease the chances of potential COIs remaining undisclosed. Even though complete COI disclosure may be a step in the right direction, it can be difficult to interpret the degree of influence these COIs have on the procedures and results of a study. Maharaj [38] attempted to solve this issue by developing a COI scale that provides a numerical score for a study based on the potential bias risk from the disclosed COIs. Scales similar to that of Maharaj could be used as a means to compare the degree of influence that disclosed COIs have in a systematic review [38]. These measures may aid in improving transparency and accessibility in medical research, dermatological and otherwise.

Our study had several strengths. The design of our analysis maximized our ability to locate and confirm potential undisclosed COIs. Our protocol was established a priori and was published for reference on Open Science Framework with other materials and protocol amendments to increase transparency and reproducibility. Data extraction was carried out as recommended by the Cochrane Handbook [39]. Prior to the study, investigators received training to account for any differences in investigational analysis, data extraction was standardized using pilot-tested Google Forms, and a search string–generating program was used to promote search uniformity. Limitations of our study include a small sample size and difficulty verifying authors with common names on patent websites. In addition, our sample lacked industry-sponsored systematic reviews, thereby preventing further analysis into the role that industry may have on the nature of reported results and conclusions. Taken together, COIs and industry sponsorship may affect the favorability of study outcomes, but the source of the discrepancy in favorability between systematic reviews with COIs and those without remains unclear [8,30,40].

COIs are common, yet often incompletely disclosed, in systematic reviews investigating melanoma interventions. However, our results suggest that the presence of author COIs did not influence the favorability of reported outcomes of melanoma systematic reviews. Future investigations are needed to more fully evaluate the influence that COIs and industry sponsorship may have on the nature and direction of results and conclusions within published dermatology literature.