Published on 03.07.20 in Vol 3, No 1 (2020): Jan-Dec
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/17370, first published Dec 23, 2019.
Evaluation of the Level of Evidence Supporting the Recommendations Constituting the American Academy of Dermatology Clinical Practice Guidelines: Cross-Sectional Analysis
Background: Clinical practice guidelines are evidence-based recommendations used by physicians to improve patient care. These guidelines provide the physician with an assessment of the benefits and harms of a treatment and its alternatives. Therefore, it is essential that the clinical practice guidelines be based on the strongest available evidence. Numerous studies in a variety of different fields of medicine have demonstrated that recommendations supported by weak evidence are a common theme in clinical practice guidelines. A clinical guideline based solely on weak evidence has the capability to reduce the quality of care provided by physicians.
Objective: Our primary objective is to evaluate the levels of evidence supporting the recommendations constituting the American Academy of Dermatology clinical practice guidelines.
Methods: Using a cross-sectional study design, authors SM and RO located all current clinical practice guidelines on the American Academy of Dermatology website on June 10, 2017, and December 11, 2019. Each recommendation and its corresponding evidence rating were extracted in a duplicate and blinded fashion. A consensus meeting was planned a priori to resolve disagreements in extractions or stratifications.
Results: In total, 6 clinical guidelines and their subsections were screened and 899 recommendations were identified. Our final data set included 841 recommendations, as 58 recommendations contained no level of evidence and were excluded from calculations. Many recommendations were supported by a moderate level of evidence and therefore received a B rating (346/841, 41.1%). Roughly one-third of the recommendations were supported by a strong level of evidence and were given an A rating (n=307, 36.5%). The clinical practice guideline with the highest overall strength of evidence was regarding the treatment of acne, which had 17 of 35 (48.6%) recommendations supported by strong evidence and only 2 (5.7%) supported by weak evidence. The clinical practice guideline with the fewest recommendations supported by strong evidence was melanoma (13/63, 20.6%).
Conclusions: Clinical practice guidelines that lack strong supporting evidence could negatively affect patient care, and dermatologists should be mindful that not all recommendations are supported by the strongest level of evidence. An increased quantity of quality research needs to be performed in the field of dermatology to improve the evidence supporting the American Academy of Dermatology clinical practice guidelines.
JMIR Dermatol 2020;3(1):e17370
Clinical practice guidelines are defined as “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” . Clinical practice guidelines are used by dermatologists to provide evidence-based treatment decisions to patients on a host of dermatologic conditions [ - ]. Evidence suggests widespread support among dermatologists regarding the use of clinical practice guidelines to improve the quality of patient care [ ]. Since many dermatologists rely on clinical practice guidelines in the clinical setting [ ], it is essential that the recommendations contained within clinical practice guidelines be supported by the best available evidence.
To date, the American Academy of Dermatology has released 6 clinical practice guidelines . The American Academy of Dermatology uses the Strength of Recommendation Taxonomy (SORT) scale to assess the strength and quality of evidence used in the development of their guideline recommendations [ ]. One advantage of the SORT scale is that it provides clinicians with a simple stratification system (A: strong supporting evidence; B: moderate supporting evidence; C: weak supporting evidence) for recommendations [ ].
Guidelines based solely on expert opinion or low-quality evidence are at a significant risk of bias [- ], a reality that can negatively affect the quality of care provided in the dermatology setting [ ]. Collectively, low-quality recommendations are a common theme in clinical practice. Shaneyfelt and Centor [ ] evaluated the strength of evidence in cardiology clinical practice guidelines and found 48% of the guidelines were based on low-quality evidence. Similar studies in dermatologic oncology [ ], comorbidities in chronic conditions [ ], orthopedic surgery [ ], fibromyalgia [ ], pediatric obesity [ ], and wound care [ ] have all demonstrated a need for improvement in the strength of evidence for clinical practice guidelines. In this study, we evaluate the strength of the evidence supporting the recommendations constituting the American Academy of Dermatology clinical practice guidelines.
Authors SM and RO located all current clinical practice guidelines from the American Academy of Dermatology website on June 10, 2017, and December 11, 2019. All 6 published guidelines and their subsections are included in this study (). We evaluated the guidelines provided by the American Academy of Dermatology as they were the first organization to develop and publish guidelines for the clinical management of various cutaneous diseases [ ]. We did not include Companion Consensus Statements, which are recommendations that return no evidence. We also excluded Appropriate Use Criteria, since these documents identify areas where sound data are not available or do not provide sufficient evidence of applying to the full range of patients seen in clinical practice [ ].
The guidelines provide a list of included studies as well as an assessment of the methodological quality of these studies. Based on the results of these assessments, a strength of evidence rating is assigned to each recommendation. These ratings are presented in each clinical practice guidelines Summary of Recommendations section. The SORT scale and definitions are included in.
SM and RO independently extracted each recommendation and its corresponding evidence rating from the clinical practice guidelines. Data extraction was conducted in duplicate fashion with each investigator masked to the other’s responses. Authors next stratified each recommendation by quality of evidence and clinical practice guideline. A consensus meeting was planned a priori to resolve disagreements in extractions or stratifications. If an agreement could not be reached, a third party arbitrator, MV, was available to resolve any disagreements. Author CC is a dermatology resident, JC is an orthopedic surgery resident, RO and TR are fourth-year medical students, SM is a student that participated in a summer research program through Oklahoma State College Center for Health Sciences, and MV is a clinical assistant professor of psychiatry and behavioral sciences.
|Guidelines and sections||Publication date||Number of recommendations|
|Atopic dermatitis (n=66)|
|Diagnosis and assessment||2014||8|
|Phototherapy and systemic agents||2014||16|
|Disease flares and adjunctive therapy||2014||16|
|Nonmelanoma skin cancer (n=97)|
|Cutaneous squamous cell carcinoma||2018||54|
|Basal cell carcinoma||2018||43|
|Office-based surgery (n=38)||2016||38|
|Phototherapy and photochemotherapy||2010/2019a||104|
aThis section includes new additional guidelines.
|Strength of recommendation||Evidence supporting recommendation||Description of recommendation|
|A||Strong||The recommendation is based on consistent and good-quality patient-oriented evidence.|
|B||Moderate||The recommendation is based on inconsistent or limited-quality patient-oriented evidence.|
|C||Weak||The recommendation is based on consensus, usual practice, opinion, disease-oriented evidence, or case series for studies of diagnosis, treatment, prevention, or screening.|
In total, 6 guidelines consisting of 899 individual recommendations were identified. Of these, 58 recommendations were not given an evidence level and were excluded from calculations, leaving 841 recommendations in our data set (). A large proportion of recommendations received a B rating, indicating moderate evidence to support the recommendation (346/841, 41.1%). Just over one-third of the recommendations were supported by strong evidence (n=307, 36.5%), while less than a quarter of the recommendations were supported by weak evidence (n=188, 22.4%; ).
Recommendations were stratified by clinical practice guideline (). The clinical practice guideline with the highest overall strength of evidence was regarding the treatment of acne, which had 17 of 35 (48.6%) recommendations supported by strong evidence and only 2 (5.7%) recommendations supported by weak evidence. The nonmelanoma skin cancer guideline fared second-best, with 42 of its 97 (43.3%) recommendations supported by strong evidence and 16 (16.5%) of the recommendations supported by weak evidence. The clinical practice guideline with the fewest recommendations supported by strong evidence was melanoma, which only had 13 of its 63 (20.6%) recommendations supported by strong evidence. The guideline with the most recommendations supported by weak evidence is office-based surgery, with half of the recommendations supported by weak evidence (19/38, 50.0%).
The years 2008 (60/60, 100%) and 2011 (16/16, 100%) had the highest percentage of recommendations supported by strong evidence (). The years 2014 (19/66, 28.8%) and 2016 (21/73, 28.8%) had the greatest percentage of recommendations supported by weak evidence. The newest guidelines, published in 2019, had a 9.6% decrease in the amount of recommendations supported by strong evidence and a 9.2% increase in recommendations supported by weak evidence. There is no consistent pattern suggesting that the level of evidence supporting the recommendations is improving or worsening.
|Guideline recommendations||Support of recommendation|
|A (Strong), n (%)||B (Moderate), n (%)||C (Weak), n (%)|
|Acne (n=35)||17 (48.6)||16 (45.7)||2 (5.7)|
|Atopic dermatitis (n=66)||15 (22.7)||32 (48.5)||19 (28.8)|
|Melanoma (n=63)||13 (20.6)||20 (31.8)||30 (47.6)|
|Nonmelanoma skin cancer (n=97)||42 (43.3)||39 (40.2)||16 (16.5)|
|Office-based surgery (n=38)||8 (21.1)||11 (28.9)||19 (50)|
|Psoriasis (n=542)||212 (39.1)||228 (42.1)||102 (18.8)|
|Total (n=841)||307 (36.5)||346 (41.1)||188 (22.4)|
The American Academy of Dermatology clinical practice guidelines aim to compile and summarize the best available evidence to guide dermatologists in their practice of evidence-based medicine. These guidelines are seen as essential by the majority of dermatologists, who believe they increase the quality of patient care . However essential these guidelines may be to dermatologists, our study found that only 36.5% of the American Academy of Dermatology published guidelines are supported by strong evidence. Furthermore, there were inconsistencies in the strength of the recommendations between the 6 guidelines, with the range of recommendations supported by strong evidence spanning from nearly 20% to 50%. The newly published 2019 guidelines actually had a lower number of recommendations supported by strong evidence compared to the guidelines published in 2016 and 2018. Based on these results, we suggest that dermatologists be mindful that not all clinical practice guidelines are based on the strongest evidence and that dermatologists should regularly review guidelines with special attention to their level of evidence.
To our knowledge, we are the first to explore the level of evidence supporting the American Academy of Dermatology guidelines to include the most recently published guidelines (2019). The results of our study further complement the paradigm demonstrated in studies of other medical specialties in which the strength of clinical practice guidelines have been examined. For example, in obstetrics and gynecology, Wright et al  found that only 30% of the recommendations were based on high-quality evidence. In the American College of Cardiologists and American Heart Association clinical practice guidelines, Tricoci et al [ ] found that a mere 11% of the recommendations were based on high-quality evidence. Shah et al [ ] found that 0% of the evidence underlying the American College of Chest Physicians clinical practice guidelines for venous thromboembolism were based on the highest quality ratings, and Poonacha et al [ ] found that only 9% of the National Comprehensive Cancer Network clinical practice guidelines recommendations were based on high-quality evidence. Members of our own research team demonstrated similar results to these studies. In addition, Meyer et al [ ] found that in the American College of Gastroenterology, only 15% of the guidelines were supported by high evidence.
These findings, as well as those of our own study, suggest that a greater emphasis be placed on conducting and completing good-quality randomized trials and other studies to improve the quality of evidence-based medicine in dermatology. For example, the discrepancy observed between the strength of evidence supporting the acne and melanoma guidelines highlights the need for more high-quality melanoma studies. The American Academy of Dermatology melanoma clinical practice guideline identifies several gaps in research, with one being a lack of randomized controlled trials, thus demonstrating the need for an interdisciplinary approach to increase the number of high-quality studies . The Strength of Recommendation Taxonomy states that for a recommendation to receive an A rating, it needs to have one of the following: a Cochrane Review with a clear recommendation, a United States Preventive Services Task Force (USPSTF) Grade A recommendation, consistent findings from at least two good-quality randomized controlled trials or diagnostic cohort studies, validated clinical decision rule in a relevant population, or a clinical evidence rating of “Beneficial” [ ]. Therefore, we recommend that further research be performed to evaluate the quality and quantity of the evidence supporting the clinical practice guidelines.
Though the American Academy of Dermatology clinical practice guideline recommendations were primarily not supported by a strong level of evidence, the recommendations (especially the acne and nonmelanoma skin cancer clinical practice guidelines) were of much higher quality than those of other fields of medicine [, , ]. Though significant work is still required to raise the quality and quantity of evidence in dermatology guidelines, dermatology seems poised to become a leader in the compilation of truly evidence-based clinical practice guidelines if this direction is prioritized.
The strengths of our study include the following: data extraction was completed in a duplicate and blinded fashion, following the recommendations set forth by the Cochrane Handbook of Systematic Reviews of Interventions ; data extraction occurred over a 3-year period to include the most recent 2019 guidelines; and the results of our study were consistent with previously published articles in numerous fields of medicine. However, our study is not without limitations. Our study evaluated only clinical practice guidelines published by the American Academy of Dermatology and therefore is not generalizable outside the field of dermatology. Additionally, our study is not generalizable toward other American Academy of Dermatology quality measures such as Appropriate Use Criteria or other published literature. As some of the guidelines were published prior to the current year, they may not accurately reflect the current levels of evidence in dermatologic literature, and therefore our study may underestimate the current research quality in dermatology. Although we evaluated the overall levels of evidence of research underpinning guideline recommendations, we did not perform risk of bias assessments on each individual study. This may be seen as a perceived limitation of this study or as an opportunity for follow-up investigations. Furthermore, it should also be noted that just because a recommendation is not underpinned by strong quality evidence does not imply that it should be omitted from clinical practice guidelines. For certain recommendations, it may be difficult or impossible to achieve strong recommendations if they are based on high-risk populations or are of a direction unlikely to receive the attention of a randomized controlled trial.
We demonstrated that the American Academy of Dermatology clinical practice guideline recommendations are primarily supported by moderate levels of evidence and only about one-third of the recommendations are supported by strong evidence. Although the American Academy of Dermatology clinical practice guidelines are supported by stronger evidence than several other guidelines, there is still a need for improvements in the quality and quantity of research in the field of dermatology.
Conflicts of Interest
- Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines. In: Graham R, Mancher M, Miller Wolman D, Greenfield S, Steinberg E, editors. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press (US); 2014.
- Emmert B, Hallier E, Schön MP, Emmert S. Disease management guidelines in dermatology: implementation, potentials and limitations exemplified by the guidelines for the management of hand eczema [in German]. Hautarzt 2011 Apr 23;62(4):308-314. [CrossRef] [Medline]
- de Haas E, de Vijlder H, van Reesema WS, van Everdingen J, Neumann H. Quality of clinical practice guidelines in dermatological oncology. J Eur Acad Dermatol Venereol 2007 Oct 09;21(9):1193-1198. [CrossRef] [Medline]
- Radtke MA, Reich K, Spehr C, Augustin M. Treatment goals in psoriasis routine care. Arch Dermatol Res 2015 Jul 31;307(5):445-449. [CrossRef] [Medline]
- Wakkee M, Lugtenberg M, Spuls P, de Jong EM, Thio H, Westert G, et al. Knowledge, attitudes and use of the guidelines for the treatment of moderate to severe plaque psoriasis among Dutch dermatologists. Br J Dermatol 2008 Aug;159(2):426-432. [CrossRef] [Medline]
- Taba P, Rosenthal M, Habicht J, Tarien H, Mathiesen M, Hill S, et al. Barriers and facilitators to the implementation of clinical practice guidelines: a cross-sectional survey among physicians in Estonia. BMC Health Serv Res 2012 Dec 13;12(1):455 [FREE Full text] [CrossRef] [Medline]
- Journal of the American Academy of Dermatology. Clinical Guidelines. URL: https://www.jaad.org/content/collection-clinical-guidelines [accessed 2019-12-22]
- Ebell MH, Siwek J, Weiss BD, Woolf SH, Susman J, Ewigman B, et al. Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature. J Am Board Fam Pract 2004 Jan 01;17(1):59-67 [FREE Full text] [CrossRef] [Medline]
- Hirsh J, Guyatt G. Clinical experts or methodologists to write clinical guidelines? The Lancet 2009 Jul;374(9686):273-275. [CrossRef]
- Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA 2009 Feb 25;301(8):831-841. [CrossRef] [Medline]
- Wright JD, Pawar N, Gonzalez JSR, Lewin SN, Burke WM, Simpson LL, et al. Scientific Evidence Underlying the American College of Obstetricians and Gynecologistsʼ Practice Bulletins. Obstetrics & Gynecology 2011;118(3):505-512. [CrossRef]
- Atkins D, Eccles M, Flottorp S, Guyatt GH, Henry D, Hill S, GRADE Working Group. Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group. BMC Health Serv Res 2004 Dec 22;4(1):38 [FREE Full text] [CrossRef] [Medline]
- Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA 2009 Feb 25;301(8):868-869. [CrossRef] [Medline]
- Boyd CM, Darer J, Boult C, Fried LP, Boult L, Wu AW. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA 2005 Aug 10;294(6):716-724. [CrossRef] [Medline]
- Kahle JT, Highsmith MJ, Kenney J, Ruth T, Lunseth PA, Ertl J. The effectiveness of the bone bridge transtibial amputation technique: A systematic review of high-quality evidence. Prosthet Orthot Int 2017 Jun 20;41(3):219-226. [CrossRef] [Medline]
- Thieme K, Mathys M, Turk DC. Evidenced-Based Guidelines on the Treatment of Fibromyalgia Patients: Are They Consistent and If Not, Why Not? Have Effective Psychological Treatments Been Overlooked? J Pain 2017 Jul;18(7):747-756. [CrossRef] [Medline]
- Nissen T, Wayant C, Wahlstrom A, Sinnett P, Fugate C, Herrington J, et al. Methodological quality, completeness of reporting and use of systematic reviews as evidence in clinical practice guidelines for paediatric overweight and obesity. Clin Obes 2017 Feb 23;7(1):34-45. [CrossRef] [Medline]
- Morton LM, Bolton LL, Corbett LQ, Girolami S, Phillips TJ. An Evaluation of the Association for the Advancement of Wound Care Venous Ulcer Guideline and Recommendations for Further Research. Advances in Skin & Wound Care 2013;26(12):553-561. [CrossRef]
- Drake LA, Yale KP, Lowery BJ, Dunbar AL, Gillies R. American Academy of Dermatology guidelines of care. Development and process. Arch Dermatol 1997 Nov;133(11):1369-1374. [Medline]
- Journal of the American Academy of Dermatology. Appropriate use criteria. URL: https://www.aad.org/practicecenter/quality/appropriate-use-criteria [accessed 2017-07-26]
- Shah S, Khan AR, Khan S, Assaly R. Evidence Underlying the American College of Chest Physicians (ACCP) Clinical Practice Guidelines for the Prevention, Diagnosis, and Management of Venous Thromboembolism. Chest 2014 Oct;146(4):821A. [CrossRef]
- Poonacha TK, Go RS. Level of Scientific Evidence Underlying Recommendations Arising From the National Comprehensive Cancer Network Clinical Practice Guidelines. JCO 2011 Jan 10;29(2):186-191. [CrossRef]
- Meyer C, Bowers A, Wayant C, Checketts J, Scott J, Musuvathy S, et al. Scientific evidence underlying the American College of Gastroenterology's clinical practice guidelines. PLoS One 2018 Oct 3;13(10):e0204720 [FREE Full text] [CrossRef] [Medline]
- Swetter SM, Tsao H, Bichakjian CK, Curiel-Lewandrowski C, Elder DE, Gershenwald JE, et al. Guidelines of care for the management of primary cutaneous melanoma. J Am Acad Dermatol 2019 Jan;80(1):208-250. [CrossRef] [Medline]
- Higgins JG. editor. Cochrane Handbook for Systematic Reviews of Interventions. John Wiley & Sons 2008;2008. [CrossRef]
|SORT: Strength of Recommendation Taxonomy|
Edited by G Eysenbach; submitted 23.12.19; peer-reviewed by T Muto, F Kaliyadan; comments to author 01.05.20; revised version received 05.05.20; accepted 14.05.20; published 03.07.20
©Courtney Cook, Ryan Ottwell, Taylor Rogers, Jake Checketts, Sanjeev Musuvathy, Matt Vassar. Originally published in JMIR Dermatology (http://derma.jmir.org), 03.07.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Dermatology Research, is properly cited. The complete bibliographic information, a link to the original publication on http://derma.jmir.org, as well as this copyright and license information must be included.