Evaluation of the Level of Evidence Supporting the Recommendations Constituting the American Academy of Dermatology Clinical Practice Guidelines: Cross-Sectional Analysis

Introduction

Clinical practice guidelines are defined as “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” [1]. Clinical practice guidelines are used by dermatologists to provide evidence-based treatment decisions to patients on a host of dermatologic conditions [2-4]. Evidence suggests widespread support among dermatologists regarding the use of clinical practice guidelines to improve the quality of patient care [5]. Since many dermatologists rely on clinical practice guidelines in the clinical setting [6], it is essential that the recommendations contained within clinical practice guidelines be supported by the best available evidence.

To date, the American Academy of Dermatology has released 6 clinical practice guidelines [7]. The American Academy of Dermatology uses the Strength of Recommendation Taxonomy (SORT) scale to assess the strength and quality of evidence used in the development of their guideline recommendations [8]. One advantage of the SORT scale is that it provides clinicians with a simple stratification system (A: strong supporting evidence; B: moderate supporting evidence; C: weak supporting evidence) for recommendations [8].

Guidelines based solely on expert opinion or low-quality evidence are at a significant risk of bias [9-12], a reality that can negatively affect the quality of care provided in the dermatology setting [3]. Collectively, low-quality recommendations are a common theme in clinical practice. Shaneyfelt and Centor [13] evaluated the strength of evidence in cardiology clinical practice guidelines and found 48% of the guidelines were based on low-quality evidence. Similar studies in dermatologic oncology [3], comorbidities in chronic conditions [14], orthopedic surgery [15], fibromyalgia [16], pediatric obesity [17], and wound care [18] have all demonstrated a need for improvement in the strength of evidence for clinical practice guidelines. In this study, we evaluate the strength of the evidence supporting the recommendations constituting the American Academy of Dermatology clinical practice guidelines.

Methods

Authors SM and RO located all current clinical practice guidelines from the American Academy of Dermatology website on June 10, 2017, and December 11, 2019. All 6 published guidelines and their subsections are included in this study (Table 1). We evaluated the guidelines provided by the American Academy of Dermatology as they were the first organization to develop and publish guidelines for the clinical management of various cutaneous diseases [19]. We did not include Companion Consensus Statements, which are recommendations that return no evidence. We also excluded Appropriate Use Criteria, since these documents identify areas where sound data are not available or do not provide sufficient evidence of applying to the full range of patients seen in clinical practice [20].

The guidelines provide a list of included studies as well as an assessment of the methodological quality of these studies. Based on the results of these assessments, a strength of evidence rating is assigned to each recommendation. These ratings are presented in each clinical practice guidelines Summary of Recommendations section. The SORT scale and definitions are included in Table 2.

SM and RO independently extracted each recommendation and its corresponding evidence rating from the clinical practice guidelines. Data extraction was conducted in duplicate fashion with each investigator masked to the other’s responses. Authors next stratified each recommendation by quality of evidence and clinical practice guideline. A consensus meeting was planned a priori to resolve disagreements in extractions or stratifications. If an agreement could not be reached, a third party arbitrator, MV, was available to resolve any disagreements. Author CC is a dermatology resident, JC is an orthopedic surgery resident, RO and TR are fourth-year medical students, SM is a student that participated in a summer research program through Oklahoma State College Center for Health Sciences, and MV is a clinical assistant professor of psychiatry and behavioral sciences.

Results

In total, 6 guidelines consisting of 899 individual recommendations were identified. Of these, 58 recommendations were not given an evidence level and were excluded from calculations, leaving 841 recommendations in our data set (Table 1). A large proportion of recommendations received a B rating, indicating moderate evidence to support the recommendation (346/841, 41.1%). Just over one-third of the recommendations were supported by strong evidence (n=307, 36.5%), while less than a quarter of the recommendations were supported by weak evidence (n=188, 22.4%; Table 3).

Recommendations were stratified by clinical practice guideline (Table 3). The clinical practice guideline with the highest overall strength of evidence was regarding the treatment of acne, which had 17 of 35 (48.6%) recommendations supported by strong evidence and only 2 (5.7%) recommendations supported by weak evidence. The nonmelanoma skin cancer guideline fared second-best, with 42 of its 97 (43.3%) recommendations supported by strong evidence and 16 (16.5%) of the recommendations supported by weak evidence. The clinical practice guideline with the fewest recommendations supported by strong evidence was melanoma, which only had 13 of its 63 (20.6%) recommendations supported by strong evidence. The guideline with the most recommendations supported by weak evidence is office-based surgery, with half of the recommendations supported by weak evidence (19/38, 50.0%).

The years 2008 (60/60, 100%) and 2011 (16/16, 100%) had the highest percentage of recommendations supported by strong evidence (Figure 1). The years 2014 (19/66, 28.8%) and 2016 (21/73, 28.8%) had the greatest percentage of recommendations supported by weak evidence. The newest guidelines, published in 2019, had a 9.6% decrease in the amount of recommendations supported by strong evidence and a 9.2% increase in recommendations supported by weak evidence. There is no consistent pattern suggesting that the level of evidence supporting the recommendations is improving or worsening.

‎View this figure

Discussion

Overview

The American Academy of Dermatology clinical practice guidelines aim to compile and summarize the best available evidence to guide dermatologists in their practice of evidence-based medicine. These guidelines are seen as essential by the majority of dermatologists, who believe they increase the quality of patient care [5]. However essential these guidelines may be to dermatologists, our study found that only 36.5% of the American Academy of Dermatology published guidelines are supported by strong evidence. Furthermore, there were inconsistencies in the strength of the recommendations between the 6 guidelines, with the range of recommendations supported by strong evidence spanning from nearly 20% to 50%. The newly published 2019 guidelines actually had a lower number of recommendations supported by strong evidence compared to the guidelines published in 2016 and 2018. Based on these results, we suggest that dermatologists be mindful that not all clinical practice guidelines are based on the strongest evidence and that dermatologists should regularly review guidelines with special attention to their level of evidence.

To our knowledge, we are the first to explore the level of evidence supporting the American Academy of Dermatology guidelines to include the most recently published guidelines (2019). The results of our study further complement the paradigm demonstrated in studies of other medical specialties in which the strength of clinical practice guidelines have been examined. For example, in obstetrics and gynecology, Wright et al [11] found that only 30% of the recommendations were based on high-quality evidence. In the American College of Cardiologists and American Heart Association clinical practice guidelines, Tricoci et al [10] found that a mere 11% of the recommendations were based on high-quality evidence. Shah et al [21] found that 0% of the evidence underlying the American College of Chest Physicians clinical practice guidelines for venous thromboembolism were based on the highest quality ratings, and Poonacha et al [22] found that only 9% of the National Comprehensive Cancer Network clinical practice guidelines recommendations were based on high-quality evidence. Members of our own research team demonstrated similar results to these studies. In addition, Meyer et al [23] found that in the American College of Gastroenterology, only 15% of the guidelines were supported by high evidence.

These findings, as well as those of our own study, suggest that a greater emphasis be placed on conducting and completing good-quality randomized trials and other studies to improve the quality of evidence-based medicine in dermatology. For example, the discrepancy observed between the strength of evidence supporting the acne and melanoma guidelines highlights the need for more high-quality melanoma studies. The American Academy of Dermatology melanoma clinical practice guideline identifies several gaps in research, with one being a lack of randomized controlled trials, thus demonstrating the need for an interdisciplinary approach to increase the number of high-quality studies [24]. The Strength of Recommendation Taxonomy states that for a recommendation to receive an A rating, it needs to have one of the following: a Cochrane Review with a clear recommendation, a United States Preventive Services Task Force (USPSTF) Grade A recommendation, consistent findings from at least two good-quality randomized controlled trials or diagnostic cohort studies, validated clinical decision rule in a relevant population, or a clinical evidence rating of “Beneficial” [8]. Therefore, we recommend that further research be performed to evaluate the quality and quantity of the evidence supporting the clinical practice guidelines.

Though the American Academy of Dermatology clinical practice guideline recommendations were primarily not supported by a strong level of evidence, the recommendations (especially the acne and nonmelanoma skin cancer clinical practice guidelines) were of much higher quality than those of other fields of medicine [10,21,22]. Though significant work is still required to raise the quality and quantity of evidence in dermatology guidelines, dermatology seems poised to become a leader in the compilation of truly evidence-based clinical practice guidelines if this direction is prioritized.

The strengths of our study include the following: data extraction was completed in a duplicate and blinded fashion, following the recommendations set forth by the Cochrane Handbook of Systematic Reviews of Interventions [25]; data extraction occurred over a 3-year period to include the most recent 2019 guidelines; and the results of our study were consistent with previously published articles in numerous fields of medicine. However, our study is not without limitations. Our study evaluated only clinical practice guidelines published by the American Academy of Dermatology and therefore is not generalizable outside the field of dermatology. Additionally, our study is not generalizable toward other American Academy of Dermatology quality measures such as Appropriate Use Criteria or other published literature. As some of the guidelines were published prior to the current year, they may not accurately reflect the current levels of evidence in dermatologic literature, and therefore our study may underestimate the current research quality in dermatology. Although we evaluated the overall levels of evidence of research underpinning guideline recommendations, we did not perform risk of bias assessments on each individual study. This may be seen as a perceived limitation of this study or as an opportunity for follow-up investigations. Furthermore, it should also be noted that just because a recommendation is not underpinned by strong quality evidence does not imply that it should be omitted from clinical practice guidelines. For certain recommendations, it may be difficult or impossible to achieve strong recommendations if they are based on high-risk populations or are of a direction unlikely to receive the attention of a randomized controlled trial.

Conclusion

We demonstrated that the American Academy of Dermatology clinical practice guideline recommendations are primarily supported by moderate levels of evidence and only about one-third of the recommendations are supported by strong evidence. Although the American Academy of Dermatology clinical practice guidelines are supported by stronger evidence than several other guidelines, there is still a need for improvements in the quality and quantity of research in the field of dermatology.