Efficacy of ChatGPT in Educating Patients and Clinicians About Skin Toxicities Associated With Cancer Treatment

Introduction

Cancer therapy often results in systemic side effects that manifest as skin toxicities [1]. While oncologists regularly interact with patients undergoing treatment, they may not possess specialized dermatological knowledge. Similarly, dermatologists may lack insights into the nuances of cancer treatment–related skin conditions. This underscores the need for a collaborative approach, to manage these complications effectively and educate patients about them. Artificial intelligence (AI) tools such as ChatGPT can enhance this effort by providing comprehensive, accessible medical information [2,3]. This study evaluates ChatGPT’s effectiveness in offering detailed information on cancer treatment–related skin toxicities, aiming to bridge the gap between patient education and medical professionals’ expertise.

Methods

Overview

We developed 22 patient-oriented and 18 oncologist-oriented questions regarding the management of cancer treatment–related skin toxicities, based on our clinical experience and research on patients undergoing cancer therapy and designed to mirror common issues observed in clinical practice. Responses to these questions were generated using ChatGPT (OpenAI) version 3.5 (Supplementary Material S1 in Multimedia Appendix 1) [4].

Three board-certified dermatologists (AL, NG, AP) specializing in oncodermatology and affiliated with a tertiary academic institution in New York City evaluated these responses. Accuracy was assessed on a scale of 1 (completely inaccurate) to 5 (completely accurate), while comprehensiveness was rated on a scale of 1 (not at all comprehensive) to 5 (extremely comprehensive). The Flesch Reading Ease Score (FRES) was interpreted on a scale of 0 (extremely difficult to read, professional level) to 100 (extremely easy to read, fifth-grade level) and calculated using an online readability tool [5]. Interrater reliability was calculated to assess the consistency of ratings across reviewers.

Ethical Considerations

This study did not involve human subjects or patient data and was therefore exempt from institutional review board approval.

Results

Accuracy scores (out of 5) averaged 4.57 (SD 0.71) for patient questions and 4.54 (SD 0.68) for oncologist questions. Comprehensiveness scores (out of 5) averaged 4.43 (SD 0.69) for patient questions and 4.37 (SD 0.80) for oncologist questions. The average FRES scores were 41.9, 47.5, and 36.0 (overall, patient, and oncologist responses, respectively), all indicating college-level comprehension. Most (13/18, 72%) oncologist responses were unanimously deemed suitable for a patient-facing educational platform (Table 1). Interrater reliability analysis for all responses demonstrated a fair level of agreement between reviewers (27.7%; Fleiss κ coefficient of 0.227; P<.001) (Table 2).

Discussion

Our findings suggest that ChatGPT holds promise as a resource for both patients and clinicians navigating the complexities of cancer treatment–related skin toxicities, given the relatively high levels of accuracy and comprehensiveness of its responses. However, the college reading level of ChatGPT’s responses poses a potential hurdle to widespread use; ChatGPT may currently be a more appropriate tool for clinicians, who will be able to comprehend its responses more uniformly compared to patients. It will likely be practically utilized by oncologists to complement their clinical judgment and that of dermatologists, particularly as AI-driven tools become increasingly integrated into clinical settings.

Reviewers identified occasional redundancies, irrelevant information, and minor inaccuracies in ChatGPT’s responses. They noted the need for its responses to be more evidence-based and to offer more up-to-date clinical recommendations when addressing oncologist questions. For instance, when responding to a question regarding the treatment of a patient experiencing rashes and blisters while taking enfortumab, ChatGPT did not recognize Stevens-Johnson syndrome as a potential concern. It only suggested “temporarily” holding the medication, even though current research recommends “permanently” discontinuing enfortumab in cases of suspected Stevens-Johnson syndrome [6]. These observations underscore that although ChatGPT generally provides useful information and could streamline its dissemination, its responses still require refinement, careful implementation, and regular monitoring to be considered for clinical use.

Integrating AI into dermatology-related patient education raises several technical and ethical considerations, including patient privacy, potential biases in AI responses, and the vital need to keep AI models current with the latest dermatology guidelines. A limitation of our study is the use of a single AI model; a comparison of ChatGPT with other models would provide a more rounded perspective on the capabilities of AI in this context.

Future research should involve incorporating additional metrics, such as clinical applicability and impact on patient outcomes, to provide a more comprehensive evaluation of ChatGPT’s potential in clinical settings. Studies with larger sample sizes, broader diversity of questions, and wider ranges of evaluators will improve our findings’ generalizability. It would be valuable to study how variations in prompt formulation affect the accuracy and comprehensiveness of ChatGPT’s responses. These enhancements would further improve ChatGPT’s ability to support both patient education and clinical decision-making in the context of cancer therapy–related skin toxicities.