This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Dermatology Research, is properly cited. The complete bibliographic information, a link to the original publication on http://derma.jmir.org, as well as this copyright and license information must be included.
Spin is defined as the misrepresentation of a study’s results, which may lead to misperceptions or misinterpretation of the findings. Spin has previously been found in randomized controlled trials and systematic reviews of acne vulgaris treatments and treatments of various nondermatological conditions.
The purpose of this study was to quantify the presence of spin in abstracts of systematic reviews and meta-analyses of melanoma therapies and identify any related secondary characteristics of these articles.
We used a cross-sectional approach on June 2, 2020, to search the MEDLINE and Embase databases from their inception. To meet inclusion criteria, a study was required to be a systematic review or meta-analysis pertaining to the treatment of melanoma in human subjects, and reported in English. We used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) definition of systematic reviews and meta-analyses. Data were extracted in a masked, duplicate fashion. We conducted a powered bivariate linear regression and calculated odds ratios for each study characteristic.
A total of 200 systematic reviews met the inclusion criteria. We identified spin in 38% (n=76) of the abstracts. The most common type of spin found was type 3 (selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention), occurring 40 times; the least common was type 2 (title claims or suggests a beneficial effect of the experimental intervention not supported by the findings), which was not present in any included abstracts. We found that abstracts pertaining to pharmacologic interventions were 3.84 times more likely to contain spin. The likelihood of an article containing spin has decreased annually (adjusted odds ratio 0.91, 95% CI 0.84-0.99). No significant correlation between funding source or other study characteristics and the presence of spin was identified.
We have found that spin is fairly common in the abstracts of systematic reviews of melanoma treatments, but the prevalence of spin in these abstracts has been declining from 1992-2020.
Skin cancer is the most common form of cancer in the United States, with more than 9500 new diagnoses each day [
The term
Notwithstanding clinicians’ reliance on systematic reviews in everyday decision-making, it has been demonstrated that reporting in the abstracts of systematic reviews is frequently flawed [
As no humans were involved in this study, it did not meet the regulatory definition of human subject research per the US Code of Federal Regulations and was not subject to institutional review board oversight. The associated protocol, extraction forms, data analysis scripts, and other study artifacts have been uploaded to Open Science Framework to ensure transparency and reproducibility [
A study team member (DW), a systematic review librarian, constructed search strategies for the MEDLINE (Ovid) and Embase (Ovid) databases and used them to locate systematic reviews and meta-analyses of treatment modalities for melanoma (
Both databases were searched from their inception. DW conducted these searches on June 2, 2020; the retrieved records were uploaded to Rayyan, a systematic review screening platform [
1. exp Melanoma/
2. (melanoma* or (pigment* adj1 cancer*) or melanocarcinoma* or nevocarcinoma*).mp.
3. 1 or 2
4. exp Therapeutics/
5. (treat* or therap* or help* or interven*).mp.
6. 4 or 5
7. 3 and 6
8. exp Melanoma/dh, dt, th [Diet Therapy, Drug Therapy, Therapy]
9. 7 or 8
10. exp ”Systematic Review“/
11. exp Meta-Analysis/
12. (”systematic review“ or ”meta-analysis“ or (systematic* adj1 review*)).ti,ab.
13. 10 or 11 or 12
14. 9 and 13
1. exp melanoma/
2. (melanoma* or (pigment* adj1 cancer*) or melanocarcinoma* or nevocarcinoma*).mp.
3. 1 or 2
4. exp therapy/
5. (treat* or therap* or help* or interven*).mp.
6. 4 or 5
7. 3 and 6
8. exp melanoma/dm, dt, th [Disease Management, Drug Therapy, Therapy]
9. 7 or 8
10. exp ”systematic review“/
11. exp meta analysis/
12. (”systematic review“ or ”meta-analysis“ or (systematic* adj1 review*)).ti,ab.
13. 10 or 11 or 12
14. 9 and 13
Studies were required to meet the following inclusion criteria: (1) a systematic review with or without a meta-analysis; (2) focused on the treatment of melanoma; (3) conducted on human subjects only; and (4) available in English. We used the PRISMA definition of systematic reviews and meta-analyses [
Before title and abstract screening commenced, authors RN and AW completed an online training course on systematic reviews and meta-analyses by Li and Dickersin [
Data were extracted in a masked, duplicate fashion using a pilot-tested Google form. Abstracts of the included systematic reviews were thoroughly examined for the presence of the 9 most severe types of spin. The 9 spin types, defined by Yavchitz et al [
The methodological quality of each study was rated as high, moderate, low, or critically low using the AMSTAR-2 scale [
The study characteristics extracted from each systematic review and meta-analysis were as follows: (1) type of intervention (surgery, pharmacologic, nonpharmacologic, combination, other); (2) date the review was received by the journal; (3) funding sources (hospital, industry, private, public, a combination of sources including industry, a combination of sources excluding industry, none, not mentioned, other); (4) whether the review discussed compliance with PRISMA or PRISMA for Abstracts [
The overall frequency of spin and its subtypes was characterized using descriptive statistics. We then used unadjusted logistic regression models to determine the binary associations of impact of extracted study characteristics on the presence of spin in the abstracts of systematic reviews and meta-analysis. We then constructed a multivariable logistic regression model to determine the influence of these variables, controlling for each, on the presence of spin. In our protocol, we prespecified the possibility of a binary logistic regression and calculated a power analysis before the start of this study to determine required sample size using GPower (version 3.1.9.7). A previous investigation of spin in abstracts of systematic reviews and meta-analyses focused on acne vulgaris suggested that spin was present in 31% of abstracts. We therefore based our power analysis on the following assumptions and parameters: (1) twenty percent of PRISMA-compliant systematic reviews and 40% of non–PRISMA-compliant systematic reviews contain spin; (2) a type I error rate of .05 (2-tailed); (3) power of .80; and (4) multiple coefficients of determination of 0.10. We thus concluded that 185 systematic reviews would be needed. These analytic decisions are documented in our protocol. We used Stata 16.1 for all analyses.
Our initial search returned 3106 unique articles, of which 718 were removed as duplicates. An additional 1972 articles were excluded during title and abstract screening. Full-text screening resulted in the exclusion of 189 articles. Thus, 227 systematic reviews met inclusion criteria and underwent random assignment, following which data were extracted from 200. Our screening (with rationale for exclusions) and randomization process is illustrated in
The most common intervention type was pharmacologic (115/200, 57.5%), followed by surgical interventions (38/200, 19%). The date range during which included systematic reviews were received by their publishing journal spanned from 1992 to 2020 (
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram demonstrating all steps of article screening with rationale provided for excluded articles.
General characteristics of systematic reviews and meta-analyses.
Characteristics | Articles (N=200) | Odds ratio (95% CI) | ||||
|
Total | Abstract contains spin | Unadjusted | Adjusted | ||
|
||||||
|
Mixed | 32 (16) | 6 (3) | 1 (Reference) | 1 (Reference) | |
|
Nonpharmacologic | 15 (7.5) | 7 (3.5) | 3.79 (0.98-14.60) | 4.69 (0.73-30.10) | |
|
Pharmacologic | 115 (57.5) | 54 (27) | 3.84 (1.46-10.02 |
2.60 (0.64-10.61) | |
|
Surgery | 38 (19) | 9 (4.5) | 1.34 (0.42-4.29) | 1.25 (0.24-6.35) | |
|
||||||
|
No | 119 (59.5) | 41 (20.5) | 1 (Reference) | 1 (Reference) | |
|
Yes | 81 (40.5) | 35 (17.5) | 1.45 (0.81-2.58) | 1.24 (0.49-3.13) | |
|
||||||
|
No | 98 (49) | 40 (20) | 1 (Reference) | 1 (Reference) | |
|
Yes | 102 (51) | 36 (18) | 0.79 (0.44-1.40) | 0.55 (0.25-1.24) | |
|
||||||
|
Not funded | 46 (23) | 15 (7.5) | 1 (Reference) | 1 (Reference) | |
|
Industry | 27 (13.5) | 14 (7) | 2.23 (0.84-5.90) | 2.08 (0.58-7.41) | |
|
Not mentioned | 86 (43) | 29 (14.5) | 1.05 (0.49-2.25) | 0.54 (0.18-1.61) | |
|
Private | 24 (12) | 8 (4) | 1.03 (0.36-2.95) | 0.74 (0.20-2.79) | |
|
Public | 17 (8.5) | 10 (5) | 2.95 (0.94-9.29) | 1.50 (0.35-6.44) | |
|
||||||
|
High | 17 (8.5) | 6 (3) | 1 (Reference) | 1 (Reference) | |
|
Moderate | 47 (23.5) | 27 (13.5) | 2.48 (0.78-7.82) | 1.83 (0.47-7.19) | |
|
Low | 19 (9.5) | 11 (5.5) | 2.52 (0.65-9.71) | 3.05 (0.60-15.48) | |
|
Critically low | 117 (58.5) | 32 (16) | 0.69 (0.24-2.02) | 0.45 (0.11-1.86) | |
5-year impact factor, mean (SD) | 6.02 (6.57) | 6.84 (7.36) | 1.03 (0.98-1.08) | 1.04 (0.98-1.10) | ||
Abstract word limit, mean (SD) | 281 (125.35) | 276 (115.84) | 1.00 (1.00-1.00) | 1.00 (0.99-1.00) | ||
Publication year (1992-2020) | N/Ac | N/A | 0.99 (0.93-1.04) | 0.91 (0.84-0.99) |
aPRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
bAMSTAR-2: A MeaSurement Tool to Assess systematic Reviews.
cN/A: not applicable.
Of 200 studies, 68 (34%) were funded, with the most common funding source being industry (27/200, 13.5%), while 46 studies were not funded (46/200, 23%) and 86 did not mention a funding source (86/200, 43%). Most studies did not mention adherence to PRISMA (119/200, 59.6%) and a total of 102 studies (51%) were published in journals whose submission guidelines recommend PRISMA adherence. The average word limit for abstracts was 281 (SD 125.35). The average 5-year impact factor for our sample was 6.02 (SD 6.57).
Among the 200 studies in our sample, we found spin in 76 (38%) of the abstracts. We frequently found more than 1 type of spin in an abstract; thus, 117 instances of spin were identified. Spin type 3—selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention—was the most common, occurring in 40 abstracts (20%;
Spin types and frequencies (%) in abstracts (N=200).
Nine most severe types of spin [ |
Abstracts containing spin, n (%) |
1. Conclusion contains recommendations for clinical practice not supported by the findings. | 4 (2) |
2. Title claims or suggests a beneficial effect of the experimental intervention not supported by the findings. | 0 (0) |
3. Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention. | 40 (20) |
4. Conclusion claims safety based on nonstatistically significant results with a wide confidence interval. | 3 (7.1)a |
5. Conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in primary studies. | 16 (8) |
6. Selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention. | 27 (13.5) |
7. Conclusion extrapolates the review’s findings to a different intervention (ie, claiming efficacy of one specific intervention although the review covers a class of several interventions). | 4 (2) |
8. Conclusion extrapolates the review’s findings from a surrogate marker or a specific outcome to the global improvement of the disease. | 13 (6.5) |
9. Conclusion claims the beneficial effect of the experimental treatment despite reporting bias. | 10 (5) |
aA total of 158 abstract conclusions did not mention safety, thus n=42.
The most severe type of spin, type 1—conclusion contains recommendations for clinical practice not supported by the findings—occurred in 4 abstracts (2%). Because 158 studies did not mention safety outcomes or safety measures in their conclusions, only 42 abstracts could be assessed for spin type 4 (3/42, 7.1%). No abstracts contained spin type 2.
From the bivariate logistic regression, the odds were 384% higher for a systematic review covering pharmacologic interventions to contain spin compared with the reference group (odds ratio [OR] 3.84, 95% CI 1.46-10.2). After adjustment for possible covariates, this association between spin and pharmacologic interventions did not remain statistically significant (OR 2.60, 95% CI 0.64-10.61). We found that the likelihood of an article containing spin has decreased annually (adjusted OR 0.91, 95% CI 0.84-0.99;
The proportion of systematic reviews containing spin in the abstract from 1992-2020.
A total of 58.5% (117/200) of systematic reviews in our sample received a methodological quality rating of “critically low” on the AMSTAR-2 scale, 9.5% (19/200) were rated “low” quality, 23.5% (47/200) “moderate” quality, and 8.5% (17/200) “high” quality. The presence of spin was not significantly associated with a study’s AMSTAR-2 rating. All AMSTAR-2 items and frequency of responses are found in
AMSTAR-2a items and frequency of responses (N=200).
AMSTAR-2 item | Response, n (%) | ||
|
Yes | No | Partial yes |
1. Did the research questions and inclusion criteria for the review include the elements of PICO (patient/population, intervention, comparison, and outcomes)? | 200 (100) | 0 (0) | 0 (0) |
2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol? | 66 (33) | 75 (37.5) | 59 (29.5) |
3. Did the review authors explain their selection of the study designs for inclusion in the review? | 103 (51.5) | 97 (48.5) | 0 (0) |
4. Did the review authors use a comprehensive literature search strategy? | 37 (18.5) | 54 (27) | 109 (54.5) |
5. Did the review authors perform study selection in duplicate? | 121 60.5) | 79 (39.5) | 0 (0) |
6. Did the review authors perform data extraction in duplicate? | 126 (63) | 74 (37) | 0 (0) |
7. Did the review authors provide a list of excluded studies and justify the exclusions? | 15 (7.5) | 65 (32.5) | 120 (60) |
8. Did the review authors describe the included studies in adequate detail? | 46 (23) | 23 (11.5) | 131 (65.5) |
9. Did the review authors use a satisfactory technique for assessing the risk of bias in individual studies that were included in the review? | 51 (28.5)b | 104 (58.1)b | 24 (13.4)b |
10. Did the review authors report on the sources of funding for the studies included in the review? | 20 (10) | 180 (90) | 0 (0) |
11. If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results? | 95 (93.1)c | 7 (6.9)c | 0 (0)c |
12. If meta-analysis was performed, did the review authors assess the potential impact of risk of bias in individual studies on the results of the meta-analysis or other evidence synthesis? | 62 (60.7)c | 40 (39.2)c | 0 (0)c |
13. Did the review authors account for risk of bias in primary studies when interpreting/discussing the results of the review? | 74 (37) | 126 (63) | 0 (0) |
14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review? | 121 (60.5) | 79 (39.5) | 0 (0) |
15. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review? | 53 (52)c | 49 (48)c | 0 (0)c |
16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review? | 163 (81.5) | 37 (18.5) | 0 (0) |
aAMSTAR-2: A MeaSurement Tool to Assess systematic Reviews.
bA total of 21 articles included only nonrandomized studies of interventions and were not included in the table, thus N=179.
cA total of 98 articles did not perform a meta-analysis, thus N=102.
Our study suggests that approximately 1 in 3 systematic reviews or meta-analyses focused on melanoma treatment modalities contain spin in their abstract. The most common type of spin identified in our sample was type 3—selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention. An example of such selective reporting occurred in a study by Verma et al [
An interesting finding was the frequency with which spin type 6 (selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention) occurred concurrently with spin type 3 (30.7%). For example, Dafni et al [
To incorporate our findings into the existing body of literature on spin, we must compare our results with previous evaluations of spin in RCTs and observational studies. Our team’s previous investigations found spin in abstracts at rates ranging from 37% in oncology RCTs [
In 2013, PRISMA released its extension for abstracts [
Our study was conducted in a fashion that maximized reproducibility and transparency. This was achieved by publishing our protocol (before the investigation’s start date), all data, and training modules to the Open Science Framework. Additional statistical reproducibility was achieved by having all data analyses confirmed by an independent group. A final strength is that data were extracted in a duplicated and masked fashion, which the Cochrane Collaboration considers to be the gold standard [
Regarding limitations, the assessment of spin is inherently subjective. To reduce subjectivity, the investigators completed several days of online and in-person training in strictly defining spin and identifying its presence. Additionally, because we searched only 2 databases (MEDLINE and Embase), some relevant studies may have been missed. Specific study characteristics had inherent limitations. For example, some studies were published before the release of PRISMA. It is unclear when journals began recommending PRISMA guidelines as previous author guidelines were not available. In addition, owing to the wide date range of published studies, we used 5-year impact factors to account for variations, which may not accurately reflect past journal impact factors. Lastly, the tool we used to appraise systematic reviews, the AMSTAR-2, was developed and published in 2017; thus, using it to rate systematic reviews published before 2017 may have resulted in lower scores.
In summary, we found spin in 38% of abstracts of systematic reviews and meta-analyses pertaining to melanoma treatment. Our results indicate that the incidence of spin in abstracts of systematic reviews focused on melanoma therapies is on par with or less than the incidence reported by investigations in other medical fields. Additionally, our results show that spin in abstracts of systematic reviews focused on melanoma therapies is decreasing. The fields of dermatology and oncology therefore have the opportunity to be leaders in reducing abstract spin prevalence and improving the quality of reporting in abstracts of systematic reviews focused on melanoma treatment.
A MeaSurement Tool to Assess systematic Reviews
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
randomized controlled trial
Development of this protocol and study was funded by the Oklahoma State University Center for Health Sciences Presidential Mentor-Mentee Research Fellowship Grant.
None declared.