Systemic isotretinoin is among the most effective treatments for moderate-to-severe acne [1]. However, its use involves significant adverse effects, strict monitoring, and complex counseling. As a result, patients frequently seek detailed information on efficacy, safety, laboratory monitoring, teratogenic risks, and long-term outcomes [2]. As health information becomes increasingly digitalized, patients are increasingly using online sources to supplement information provided by physicians [3]. AI fulfills diverse roles in dermatology, such as facilitating diagnosis, monitoring disease progression, guiding treatment selection, providing patient information, and supporting medical education [4,5]. Numerous studies across various medical specialties have used AI language models to evaluate responses to patient questions. However, standardized guidelines for performance assessment are lacking, and the methodological details of these studies often remain unclear [6]. Several studies have assessed the performance of large language models (LLMs) in responding to patient questions about acne and systemic isotretinoin, with mixed results for accuracy and quality [7-11]. Due to isotretinoin’s teratogenicity, monitoring needs, and medicolegal considerations, it is essential that the information provided is both accurate and easily understandable.

The study aimed to comparatively evaluate ChatGPT and Gemini in their responses to patient questions regarding systemic isotretinoin, assessing performance across multiple quality domains, interrater reliability, and readability metrics.

In this study, ChatGPT demonstrated significantly higher performance across evaluation metrics, including accuracy, completeness, clarity, and overall quality, while Gemini showed greater interrater reliability and superior readability. Accordingly, the choice of LLM may depend on whether the priority is high-quality medical content or improved patient comprehension.

Recent evaluations indicate that LLMs generate more contextually appropriate medical responses than traditional internet-based resources [13]. ChatGPT has demonstrated potential as a public dermatology resource, particularly for common dermatological diseases, with 88% of its responses rated as appropriate [14]. However, the clinical utility of LLMs remains constrained by their propensity to generate fictitious citations [15,16]. Their inability to account for individual patient factors and age-related nuances may also lead to misleading guidance, potentially resulting in adverse outcomes based on unverified data [11].

The literature further demonstrates that LLM accuracy and comprehensiveness ratings vary considerably across dermatologic diseases and topics [11]. For example, ChatGPT’s accuracy has ranged from 36% to 90% across various topics [6,14,17,18]. ChatGPT showed higher accuracy in dermatological topics with well-established evidence, such as melanoma and nonmelanoma skin cancers, rosacea, and alopecia. In contrast, lower accuracy was observed in areas requiring individualized approaches or where scientific consensus remains limited, including the effects of diet on skin diseases, nonpharmacological treatments for eczema and psoriasis, antiaging interventions, skincare routines, and wound healing [14]. In a study examining isotretinoin-related queries, ChatGPT’s accuracy varied by domain: daily life achieved 72% reliability, while the treatment process yielded 64% with outdated or incorrect data [7]. In contrast, our findings showed no significant differences across query categories.

Our findings contradict previous research indicating that Gemini provided more accurate answers than ChatGPT on isotretinoin questions [8], as well as studies asserting Gemini’s superiority in comprehensiveness in melanoma-related queries [18]. Recent evidence regarding Mohs surgery indicates that GPT-4 outperforms Gemini (Google Bard) in clinical relevance and accuracy, providing answers that closely align with official American College of Mohs Surgery recommendations [19], suggesting that differences in topic may significantly influence the performance of LLMs. Furthermore, such discrepancies may result from the absence of standardized evaluation criteria and differences among LLM models and versions.

Consistent with prior research on isotretinoin, which reported a low Fleiss κ (κ=0.045) [7], our study also demonstrated similarly weak interrater agreement. Consistent with our findings, Gemini demonstrated higher interrater agreement than ChatGPT in a study of common dermatological conditions [20]. This low interrater agreement may be partially attributed to heterogeneity in physicians’ clinical communication styles and expectations in patient education. To address inconsistent evaluations, it has been proposed that LLM outputs be assessed using standardized, guideline-based frameworks [21]. Nevertheless, the complex dynamics of physician-patient interactions and the subjective nature of clinical judgment continue to influence evaluation outcomes.

Previous studies suggest that AI-generated health information often exceeds recommended readability levels, potentially limiting accessibility for individuals with low health literacy [21-23]. Recent studies have shown that successive iterations of LLMs exhibit progressive improvement in generating more readable and accessible content [24,25]. In our study, both ChatGPT and Gemini produced patient information at readability levels exceeding recommended thresholds for public education materials. Consistent with our results, some studies report higher readability for Gemini [18], whereas others find no significant readability differences between Gemini and ChatGPT [8]. High readability demands, incomplete incorporation of updated approaches, and limited actionable guidance preclude the use of LLM outputs as standalone patient education tools [7,8]. For high-risk medications such as isotretinoin, patients should be explicitly informed that LLM-generated outputs constitute preliminary information and do not substitute for professional medical guidance. Given the substantial patient concerns about isotretinoin [26-28], clear and accessible educational materials may mitigate misconceptions, reduce treatment resistance, and improve adherence.

Additionally, ChatGPT can be directed to create patient brochures at predefined reading levels, thereby improving accessibility for diverse patient populations [18,29]. Effective prompt engineering enables LLMs to deliver clearer, more targeted responses [8,30-32]. Incorporating an audience-adaptive response model in which users identify as health care professionals or laypersons prior to output generation could facilitate flexible adjustment of terminology and grade-level complexity.

Gemini consistently advised consultation with a health care professional, whereas this was not observed in our study with ChatGPT, despite prior reports of such disclaimers [33,34]. This suggests that Gemini may offer a comparatively safer profile for patient safety.

The strengths of this study include methodological transparency, a comprehensive multidimensional evaluation approach, a comparative design, and the provision of actionable recommendations. This study has several limitations. The selected questions may not accurately represent real-world patient inquiries. The study was conducted exclusively in English, and language-dependent variability in AI model training data and source retrieval may limit generalizability across different languages. Moreover, despite blinded assessment, clinicians familiar with AI language models may have inferred the model’s identity, potentially introducing observer bias. Another limitation is that our analysis included only ChatGPT and Gemini. Therefore, the findings may not be generalizable to other commercially available or open-source LLMs. Given the rapid evolution of LLMs, the findings of this study reflect the performance of GPT-4o and Gemini 2.0 Flash at the time of evaluation. As newer model iterations are released with enhanced reasoning and language capabilities, ongoing comparative studies are required to determine whether these advancements yield measurable improvements in clinical quality, reliability, and readability. Furthermore, because LLMs are frequently updated, their responses and performance may change over time. Future studies should include a broader range of models and evaluate changes across model versions longitudinally.

In conclusion, there is a clear distinction between ChatGPT’s technical accuracy and Gemini’s linguistic accessibility. For these models to function as reliable clinical tools, transparent source attribution and patient-centered, intelligible outputs are imperative. Adopting standardized, objective evaluation frameworks is also crucial for enabling robust comparisons of outputs and tracking model performance over time, thereby mitigating interrater variability. However, the complexity and heterogeneity of patient characteristics make it unlikely that clinicians will be fully substituted for by AI, particularly when it comes to contextualizing clinical evidence for individual patients.