ChatGPT was selected for its popularity and prior evidence of superiority in dermatology-related tasks. A double-blind study found dermatologists preferred ChatGPT over Google’s Bard for patient handouts [7]. ChatGPT 4o was used. DermGPT’s only available model was used.

Two dermatology residents, CZ and NMG, authored a list of questions posed to each LLM (Multimedia Appendix 1). Three questions to which DermGPT did not provide a response were excluded as nonevaluable item pairs. The two models’ answers for a given question were paired and assigned as A or B using a computer-generated randomization list. Any identifiable metadata such as formatting was cleared. The survey was distributed to dermatologists at the University of California, Irvine, and the University of California, Davis, via email and QR codes. Survey takers were informed that both responses were produced by LLMs, but they were blinded to which model produced which response. They were asked to choose their preferred answers based on quality—specifically, which answer they thought would be best suited for patient care or was most accurate.

The rating options were as follows:

Statistical analysis was conducted using SAS OnDemand for Academics (version 9.4). χ² tests (P<.05) assessed significance. Interrater reliability was not prespecified and not assessed; ratings were aggregated at the item level.

This study used a voluntary, anonymous survey of physicians and residents. According to institutional and national guidelines, the project did not require institutional review board review because no identifiable information was collected and the study posed minimal risk.

Participants provided implied consent by completing the survey after being informed of its purpose and their ability to withdraw at any time. No compensation was provided. The survey responses were analyzed in aggregate to ensure anonymity and privacy in accordance with institutional standards. The study followed the ethical principles of the Declaration of Helsinki, adhered to Committee on Publication Ethics guidelines, and met all institutional requirements for minimal-risk survey research.

Of 64 dermatology faculty and 30 residents across the University of California, Irvine, and the University of California, Davis, we received a total of 19 responses, comprising 13 attending physicians and 6 residents or fellows. This corresponds to an overall response rate of approximately 20%.

Overall, DermGPT’s answers (48.1%) were preferred over ChatGPT’s (28.4%); the χ² test was significant with P=.04 (P<.05). In the attending group, DermGPT’s answers were preferred (93/195, 47.7%) over ChatGPT’s (56/195, 28.7%). Likewise, in the resident group, DermGPT’s answers were preferred (44/90, 48.9%) versus ChatGPT (25/90, 27.8%) (Table 1).

Overall, ChatGPT references (46%) were preferred over DermGPT (23.5%; χ²₂=1.385; P=.50). In the attending group, ChatGPT references were also preferred (94/195, 48.2%) over DermGPT (45/195, 23.1%). Likewise, in the resident group, ChatGPT references were preferred (37/90, 41.1%) versus DermGPT (22/90, 24.4%) (Table 2).

Out of 195 responses, users generally preferred DermGPT’s answers, while ChatGPT was favored for its reference citations (Table 2). DermGPT’s concise and well-phrased responses made it accessible for quick clinical reference. However, 3 questions were excluded because DermGPT issued disclaimers instead of direct answers, recommending consultation with a dermatologist or guidelines. The multimedia appendices show the results tabulated from SAS as well as the questionnaire and responses (Multimedia Appendices 1-6).

ChatGPT consistently cited reputable references such as the Journal of the American Academy of Dermatology and the Journal of the American Medical Association, contributing to user trust and perceived academic rigor. Although DermGPT offers clarity, ChatGPT’s strong sourcing enhances credibility. These results suggest the potential for a hybrid model that combines both strengths.

Our study was constrained by a small rater sample (n=19) and multiple ratings per rater and per question. As a result, P values should be interpreted as exploratory rather than confirmatory. The sample may not represent all dermatology clinicians, limiting generalizability. Subgroup patterns were underpowered.

Several studies have compared LLMs to each other and to humans. He et al [8] found GPT-4 sometimes produced inaccurate, nonindividualized responses to laboratory-related queries. Iannantuono et al [9] compared ChatGPT-4, ChatGPT-3.5, and Google Bard in immunooncology, stressing the need for expert verification. Fernández-Pichel et al [10] found LLMs answered 80% of health questions accurately, though results were sensitive to prompt phrasing. This is the first study comparing ChatGPT and DermGPT for dermatologic responses.

Future research should include models like Claude and Gemini, expand sample size, and explore combining DermGPT’s brevity with ChatGPT’s sourcing. These results highlight the importance of balancing clarity and citation in artificial intelligence–assisted medical tools.