Although many skin cancers are preventable, there has been a global increase in skin cancer incidence in recent years. Skin cancer accounts for one-third of all cancer diagnoses of the nonmelanoma type, with basal cell carcinomas accounting for 75%. Although melanoma represents only 2% of all skin cancer cases, it is responsible for 80% of skin cancer–related deaths. The annual incidence of melanoma has increased by 4% especially among fair-skinned populations in North America, northern Europe, Australia, and New Zealand [1-3]. In the United States, melanoma is the fifth most common cancer, with an increase in incidence of approximately 320% since 1975 [4]. Globally, melanoma cases are projected to increase from 325,000 in 2020 to approximately 510,000 by 2040—a 50% rise—and mortality is expected to increase by 68% over the same period.

This trajectory underscores the urgent need for effective sun protection strategies and enhanced public health education. UV radiation is the primary modifiable risk factor for skin cancer, causing direct DNA damage to skin cells [5]. Despite growing awareness, adherence to comprehensive sun protection guidelines remains suboptimal [6], creating an education gap that emerging technologies may help address.

Given the documented gap between guideline recommendations and public adherence, artificial intelligence (AI) platforms such as ChatGPT represent a new opportunity to improve health communication and patient education at scale. ChatGPT enhances patient education by providing clear and accessible information about dermatological conditions, improving patient understanding and engagement [7]. However, a pilot study by Mondal et al [8] raised significant concerns regarding the accuracy of AI-generated content and potential issues related to text similarity. More broadly, evaluating conversational health AI tools requires frameworks that go beyond surface-level correctness to assess evidence grounding, clinical contextual adequacy, and usefulness to lay users [9].

This study aimed to evaluate the quality of responses generated by ChatGPT (OpenAI) regarding sun protection and skin cancer prevention by systematically comparing them to American Academy of Dermatology (AAD) guidelines. Using an author-developed assessment framework covering 4 domains—accuracy, completeness, clarity, and relevance—this descriptive content analysis sought to characterize ChatGPT’s utility and limitations as a source of patient-facing dermatological guidance.

The findings will inform patient education strategies and contribute to evidence-based guidelines for AI integration in health care communication. This research is particularly timely given the growing reliance on AI-powered health information resources and their potential impact on public health behaviors.

This was a descriptive content analysis comparing AI-generated responses to common patient questions on sun protection and skin cancer prevention against established clinical guidelines. Nine specific questions were developed to cover sunscreen use, protective clothing, and early skin cancer detection, reflecting common public inquiries consistent with topics addressed in previous evaluations of AI-generated dermatological content [10].

A structured summary of the study design following the METRICS (model, evaluation, timing, range/randomization, individual factors, count, and specificity of prompts and language) framework for standardized reporting of generative AI studies in health care [11] is shown in Table 1.

The following nine questions were presented to ChatGPT: (1) “How to prevent skin cancer?” (2) “What sunscreen should I use?” (3) “How much sunscreen should I use, and how often should I apply it?” (4) “When should I use sunscreen?” (5) “Is a high-number SPF better than a low-number one?” (6) “What to wear to protect my skin from the sun?” (7) “What are the common types of skin cancer?” (8) “How to perform a skin self-exam?” (9) “What are the ABCDE warning signs of melanoma?”

To maintain consistency, questions were submitted verbatim without modifications or additional context, and responses were documented exactly as provided without interpretation. The full responses are shown in Table 2.

All 9 questions were submitted within a single uninterrupted session on May 13, 2025, using the free-tier ChatGPT interface (GPT-4). The ChatGPT memory function and conversation history were enabled throughout the session and were not reset between questions, meaning that the model retained the full conversational context as subsequent questions were posed. This reflects the typical use pattern of a lay public user and was intentional in capturing the realistic output that a patient would receive in a real-world interaction. However, it represents a methodological limitation affecting response independence as later responses may have been influenced by earlier conversational context (this is discussed further in the Limitations section). The exact model subversion could not be pinned on the free-tier interface; GPT-4 was the version designated to free users at the time of data collection. A single response was collected per question, consistent with a realistic single-query patient use scenario [10].

ChatGPT’s responses were evaluated against established AAD guidelines using an author-developed 5-point ordinal rating scale covering 4 domains:

A 5-point scale was applied to each domain: 0=“completely inadequate or incorrect”; 1=“poor, significant inaccuracies or omissions”; 2=“fair, partially correct with notable gaps”; 3=“good, mostly correct with minor gaps”; 4=“very good, only minor omissions”; and 5=“excellent, fully correct and comprehensive.”

Two board-certified consultant dermatologists, each with over 15 years of clinical experience—one based in Jordan (HQ) and one based in the United Kingdom (AA-R)—independently evaluated all ChatGPT responses across the 4 domains. Scoring was completed independently without communication between raters. Score sheets were then compared; where disagreements were identified, the 2 raters discussed each item until consensus was reached without involvement of a third arbitrator. For 2 disagreements (questions 3 and 8 in the “completeness” domain), consensus was reached by averaging scores; for one disagreement (question 5 in the “completeness” domain), rater 2 (AA-R) deferred to rater 1 (HQ) after discussion. Consensus scores are reported throughout.

Interrater reliability (IRR) was assessed prior to consensus resolution using SPSS Statistics (version 29; IBM Corp). The linear weighted Cohen κ was calculated to account for the ordinal nature of the scale, with weights proportional to the magnitude of disagreement. The intraclass correlation coefficient (ICC) was computed using a 2-way mixed-effects model, absolute agreement, single rater (ICC(2,1)). κ values were interpreted as slight (<0.20), fair (0.21‐0.40), moderate (0.41‐0.60), substantial (0.61‐0.80), and almost perfect (>0.80) agreement. An ICC of 0.75 or higher was considered to indicate excellent reliability [12].

Descriptive statistics were calculated for each evaluation domain: mean, median, SD, IQR, minimum, and maximum. Given the small sample size (n=9 questions) and ordinal scale, median and IQR values are reported alongside mean and SD values. Domains achieving a mean of 5.0 with an SD of 0.0 reflected ceiling effects and were interpreted accordingly. All analyses were performed using SPSS Statistics (version 29).

This study did not involve human participants or patient data; therefore, institutional review board approval was not required. As this study involved only the analysis of AI-generated text against published clinical guidelines, no ethical concerns regarding human participants apply. This study was not a randomized clinical trial; accordingly, trial registration and CONSORT (Consolidated Standards of Reporting Trials) reporting requirements do not apply.

Preconsensus rating data for all 36 item-domain combinations (9 questions × 4 domains) are shown in Table 3. Raters agreed exactly on 91.7% (33/36) of the items. All 3 disagreements occurred within the “completeness” domain and were of a magnitude of –1 point to +1 point (question 3: rater 1=4 and rater 2=5, resolved by averaging to 4.5; question 5: rater 1=4 and rater 2=3, resolved by rater 2 deferring to rater 1 [score of 4]; and question 8: rater 1=5 and rater 2=4, resolved by averaging to 4.5). No disagreements occurred in the “accuracy,” “clarity,” or “relevance” domains. The linear weighted Cohen κ was 0.80, indicating substantial to almost perfect interrater agreement. The ICC(2,1) was 0.85, confirming excellent reliability.

ChatGPT’s responses were systematically evaluated using a 5-point ordinal rating scale across 4 domains. Descriptive statistics for each domain are summarized in Table 4, and question-level scores across all four domains are presented in Table 5.

Accuracy and clarity both demonstrated ceiling effects, with all 9 responses receiving scores of 5.0/5 and zero variance (SD 0.0; IQR 0.0). This indicates that, within the scope of the 9 questions evaluated, ChatGPT produced no factually incorrect statements. Relevance also scored highly (median 5.0, IQR 0.0; mean 4.9, SD 0.3), indicating that the responses consistently addressed the question asked with clinically applicable content. Completeness was the weakest domain (median 4.0, IQR 0.5; mean 4.1, SD 0.6; range 3.0‐5.0), with the greatest variability between questions. The lowest-scoring response was that to question 6 (sun-protective clothing; mean 4.25 overall), which also had the lowest completeness score (3.0/5) and the only below-perfect relevance score (4.0/5). Errors across all domains were predominantly omissions rather than factual inaccuracies.

This descriptive content analysis demonstrates that ChatGPT (GPT-4) generates largely accurate and guideline-consistent responses to common patient questions on sun protection and skin cancer prevention, with an overall mean score of 4.75/5 against AAD recommendations. The primary area of weakness was completeness (mean 4.1/5, SD 0.6), reflecting a consistent pattern of errors by omission rather than commission. This distinction is clinically meaningful: ChatGPT did not produce factually incorrect information in this analysis but delivered incomplete guidance in most responses, with potential real-world implications for patient behavior that depend on the specific omissions.

ChatGPT consistently provided accurate advice across all 9 questions, with no factual errors identified. Questions concerning sunscreen use, application frequency, and melanoma warning signs received the highest combined scores, indicating reliable performance on specific, well-defined topics. Questions requiring comprehensive, multicomponent answers—particularly regarding sun-protective clothing (question 6)—received the lowest completeness score (3.0/5) and the lowest overall mean score (4.25/5).

The clinical significance of these completeness gaps should not be underestimated. A patient who receives ChatGPT’s response to question 6 and acts on it exclusively would not be warned that wet clothing provides substantially reduced UV protection, would not know that mesh and ventilated garments may have lower UPF ratings than they appear, and would not receive guidance on protective footwear. For a fair-skinned patient with a high risk profile, such omissions could translate into inadequate sun protection despite perceived compliance. Similarly, the omission of guidance on protecting children’s skin (question 1) is particularly important given that childhood UV exposure is a critical determinant of lifetime melanoma risk.

These findings are consistent with those of prior research demonstrating that ChatGPT provided clinically appropriate recommendations for 96% of core questions in comparable studies, with SPF guidelines and sun-protective clothing among the topics evaluated [10]. The predominance of omission over commission errors in our study aligns with the broader pattern across large language model (LLM) evaluations in clinical medicine, wherein AI-generated content is generally accurate but selectively incomplete.

An important methodological consideration is that surface-level guideline alignment and clinical usefulness are not equivalent. A response may receive a high accuracy score yet remain incomplete, inadequately contextualized, or potentially misleading for patients with specific risk profiles or geographic constraints. Product-specific sunscreen recommendations, for example, may be accurate in the context of widely available products but inaccessible in other health care settings. This distinction—between what is stated being correct and what needs to be stated being complete—should inform how AI-generated health content is evaluated in future research and underscores the importance of using multi-domain evaluation frameworks rather than accuracy alone [9]. Recent work evaluating LLMs for clinical classification tasks has similarly demonstrated that outputs may be technically correct yet clinically insufficient for complex or context-dependent decisions [15].

Previous research has demonstrated that AI-driven interventions such as skin aging simulations have produced long-term improvements in sun protection behaviors, particularly among younger adults [20]. This suggests that AI tools such as ChatGPT could serve not only as information sources but also as platforms that motivate adherence to sun protection recommendations. Consistent with a human-centered framework for AI in health care and education, ChatGPT is most appropriately positioned as an adjunct that supports and extends clinician-led patient education rather than a stand-alone substitute for professional guidance [12]. Patient-facing AI tools are most effective when integrated into a care model that maintains physician oversight and encourages users to seek professional confirmation of AI-generated advice.

While this study focused on preventive education, AI is also being used in clinical dermatology for diagnostic support. A recent review demonstrated that AI can accurately analyze clinical and dermoscopic images, sometimes surpassing human experts in specific settings, but emphasized that the best outcomes require combining AI tools with clinical expertise [21].

This study evaluated ChatGPT (GPT-4) exclusively, which was the most widely available and most extensively studied conversational AI tool for health information at the time of data collection (May 2025). Other platforms—including Google Gemini, Grok, and Anthropic’s Claude—may generate meaningfully different responses to the same prompts and have not been benchmarked against AAD guidelines in this context. Multi-platform comparative evaluation represents an important direction for future research.

Several limitations should be acknowledged. First, the 9 questions constitute a targeted content analysis rather than a comprehensive evaluation of ChatGPT’s dermatological knowledge scope; follow-up questions, ambiguous prompts, and unsafe queries were not assessed. Second, all questions were submitted in a single session, with memory enabled and conversation history not reset between questions, meaning that later responses may have been influenced by prior conversational context. Third, a single output was collected per question; given the stochastic nature of LLMs, responses may vary across sessions, and findings reflect a single–time point snapshot. Fourth, ChatGPT’s outputs are subject to change as underlying model versions are updated, and findings from May 2025 may not be reproduced with future versions. Fifth, 2 raters were used; although IRR was excellent (weighted κ=0.80; ICC=0.85), a larger panel would further strengthen scoring validity. Sixth, the evaluation used an author-developed rating scale without formal psychometric validation; future studies should develop and validate a standardized instrument. Seventh, the comparison was limited to AAD guidelines; international guideline variations were not assessed. Eighth, all prompts were submitted in English; because prompt language has been shown to critically shape the readability, completeness, and terminology of AI-generated dermatological content, responses to equivalent prompts in other languages may differ, and our findings may not generalize to non–English-speaking populations [22]. Finally, product-specific brand recommendations in ChatGPT responses may reflect products not available in all geographic regions.

This descriptive content analysis demonstrates that ChatGPT generates largely accurate and guideline-consistent advice on sun protection and skin cancer prevention, with responses closely aligned with AAD recommendations across all 4 evaluated domains. Errors were predominantly omissions rather than factual inaccuracies, with completeness representing the primary area for improvement. The clinical significance of these omissions varies by question but warrants attention, particularly for high-risk patient groups. Although ChatGPT cannot replace professional medical advice, it can serve as a valuable adjunct tool for public health education—most effectively when integrated within a clinician-led educational framework that ensures contextual appropriateness and guideline completeness [12]. With appropriate, ongoing evaluation, AI-assisted dermatological education has significant potential to support skin cancer prevention strategies.