This randomized controlled trial used block randomization in blocks of 4, using a web-based randomizer [32] to generate a random number sequence of numbers 1 through 4. Participants were allocated continuously and sequentially as they signed up to either “Intervention” or “Control” with a 3:1 ratio, as we anticipated a large effect size of our primary outcome and desired further power for the analysis of data from the intervention group. Study recruitment was open for 30 days from mid-November to mid-December 2021.

Eligible PCPs (doctors currently working in the primary care sector) were recruited at a conference in November 2021 (Lægedage) in Denmark by the speakers at 3 skin cancer and melanoma sessions. Interested physicians scanned a QR code and signed up to receive information material and an invitation to participate in the study by email, including a link to a web-based survey (Google Forms, Google Ireland Limited, 2022) which contained a consent form. The survey also included questions about their experience with and use of dermoscopy, including their confidence in their diagnostic abilities on a scale from 1 (low) to 4 (high), and ended with a skin cancer multiple choice questionnaire (MCQ) including 12 patient cases from a list of 25 skin lesion cases with previously established validity evidence [1]. The maximum number of points acquired on the test is 12, indicating high diagnostic accuracy.

Participants in the intervention group were invited to one of several web-based initiation meetings where they were instructed on how to download, install, sign up, and access the LIIR through a quiz-based smartphone app for practicing skin cancer diagnostics. Participants could also download and install the app independently using a pdf-guide. After installation, the participants in the intervention group were given 8 days of access from the day of their sign-up in the app, in which they were asked to diagnose 500 digital patient cases. They were sent email reminders on days 4, 7, and 8. After the 8 days, they were told to abstain from using the app for 8 days (washout period) before answering a final skin cancer MCQ with 12 new cases.

Participants in the control group continued their clinical practice as usual. They were not given access to the LIIR nor received any intervention before they completed the final skin cancer MCQ 16 days after their initial MCQ. See the study diagram in Figure 1.

The trial’s primary outcome was the participants’ score on a skin cancer MCQ before and after the intervention, with both per-protocol and intention-to-treat analysis.

Secondary outcomes included the progression of the participants’ ability to correctly diagnose and classify digital patient cases in the LIIR across the intervention period, descriptive analysis of which diagnoses were most commonly misclassified and misdiagnosed, change in the participants’ diagnostic confidence, and the average time spent training.

Participants were unaware of their allocation until they had answered the initial questionnaire and MCQ. No measurements were taken to blind the principal investigator when receiving questionnaire responses and MCQ test results or when performing the statistical analysis comparing the 2 groups.

For the educational intervention in this study, we used the educational mobile app Dermloop Learn (Melatech ApS) [33] and its LIIR that has 3 main functionalities: Quizzes with anonymized digital patient cases, written learning modules, and user tracking.

The app used a library of 2376 digital patient cases with a diagnosis (either confirmed by histopathology or clinical consensus of 2 or more clinicians) belonging to 1 of 7 diagnosis groups (nevus, seborrheic keratosis (SK) or solar lentigo, dermatofibroma, hemangioma, melanoma, basal cell carcinoma or squamous cell carcinoma) where any subtype was considered correctly diagnosed if the participant answered the diagnosis-group correctly (eg, “Melanoma” was correct for both superficial spreading melanoma and lentigo maligna). Each digital patient case included the age and gender of the patient, a clinical and a dermoscopic image of the lesion, and its location on a 3D avatar; all of which are referred to as a “case.” See Figure 2 for an overview and example.

The participants trained in pattern recognition (quizzing) with sessions containing 10 cases selected from the library. One case was presented at a time with a clinical image, dermoscopic image, and the lesion’s location. The participant selected benign (and picked the suspected diagnosis from among nevus, SK or solar lentigo, dermatofibroma, or hemangioma) or malignant (and picked the suspected diagnosis from among melanoma, basal cell carcinoma, or squamous cell carcinoma). Immediately after answering a case, the user got feedback on whether the answer was correct or not, what the correct answer was, and an option of being taken to a written learning module on the correct and incorrect answer, respectively. The smartphone app has written learning modules on 36 diagnoses and subdiagnoses corresponding to the diagnoses of the cases contained in the LIIR. Each written learning module has an introductory section followed by sections on histopathology, clinical presentation, dermoscopic features, and differential diagnoses to the diagnosis, all including illustrations or examples from the LIIR.

As the user answered cases, the app compiled a list of the diagnoses for which the individual user had the most difficulty giving correct answers. The list was shown on the front page of the app (Overview page in Figure 2), nudging the user towards reading the corresponding written learning modules. The app also tracked how many cases each user had diagnosed, their answer to each case, how long they spent with each case, what written modules they opened, and how long they spent reading each time they opened a written module.

The Dermloop Lean app underwent no changes, updates, or bug fixes and there was no downtime during the trial period.

Sample size and power calculations were done by assuming a 50% diagnostic accuracy at baseline. Based on the preliminary results of a previous study using the same intervention on medical students [31], we anticipated a 20% (SD 15%) effect of the educational intervention, which should let us show a difference between the groups by including 96 participants with a 3:1 allocation ratio with a statistical significance level of .05 and 80% power.

Based on the methods of the similar and recent study by our research group [31] using Generalized Estimating Equations, learning curves of the participants’ probability to diagnose cases correctly were expressed as linear splines with a single knot at 100 cases using a logistic regression model with random intercept with correct diagnosis or not as an outcome. A similar model was made with the correct classification of each case as malignant or benign as the outcome. The participants’ time spent reading and quizzing was summed separately for each 100-case period.

MCQ scores for both groups before and after the study period were compared using 2 tailed Welch 2 Sample t test.

In this study, the intention-to-treat analysis included those individuals who participated and also answered the final questionnaire. The per-protocol analysis included those individuals who completed the stipulated 500 cases.

Statistical analyses were performed in R [34] (version 4.2.0; R Foundation for Statistical Computing) using GEE R Pack [35,36] for the primary analysis and learning curves and Excel (Microsoft Corp) [37] for descriptive statistics of study questionnaire responses.

The study was conducted in accordance with the principles of the Declaration of Helsinki. All participating doctors were given oral and written information about the study. All participants were informed that filling out the survey was seen as consent to participate. There was no financial compensation for the participants in the study. The study intervention is an educational intervention for medical doctors with no patient participation which does not require ethics board review [38].

A total of 279 people applied for more information about the study, 135 accepted the invitation and finished the initial questionnaire, of which 7 were excluded as they were not doctors from the primary care sector. Initially, 96 were randomized to the intervention and 33 to the control group. From the intervention group, 84 subsequently set up a user profile granting them access to the smartphone app and its LIIR, of which 82 answered the final MCQ and were included in the intention-to-treat analysis (see CONSORT [Consolidated Standards of Reporting Trials] diagram in Figure 1). Demographics on all participants and their experience with dermoscopy can be found in Tables 1 and 2. The intervention group completed 467 cases on average (median 505, range 0-1679), and 54 participants completed 500 cases (as per protocol) or more and were included in the Per-Protocol analysis. All 33 participants from the control group finished the final MCQ. The time from the initial to the final MCQ was 21 and 19 days for the intervention and control groups, respectively.

The average MCQ score of the PCPs in the intervention group (intention-to-treat analysis) improved from 52.0% (6.2 correct answers out of 12) to 66.4% (8.0 correct answers out of 12); an improvement of 14.3 percentage points (95% CI 9.8-18.9; P<.001). Those participants who diagnosed 500 patient cases or more (per-protocol analysis) on average improved from 53.9% (6.5 correct answers out of 12) to 70.5% (8.5 correct answers); an improvement of 16.6 percentage points (95% CI 11.3-22.0; P<.001). The MCQ score of the control group did not improve during the study period (P=.94). See Figure 3 for a box-and-whiskers plot of each group’s initial and final MCQ percentages.

A post hoc analysis of the participant’s initial MCQ score and their years of experience diagnosing skin lesions found no correlation, as shown in Figure 4.

Analysis revealed a statistically significant difference in MCQ scores between participants who diagnosed less than the protocolled 500 cases and participants who diagnosed 500 or more (1.5 points, P<.001; 95% CI 0.7-2.3), and post hoc analysis found similar results between participants who diagnosed less or more than 200 cases (1.5 points, P=.004; 95% CI 0.5-2.4), respectively, as depicted in Figure 5. There was no statistically significant difference between the MCQ scores of those participants that diagnosed between 449-549 cases and those that did 550 or more (P=.12).

The participants in the intervention group performed a total of 39,022 diagnostic evaluations during the study period. When comparing the participants’ answers on the first (case 0-50) and last (case 451-500) 50 cases, we found that their probability of correctly classifying a case as benign or malignant increased by 6.6 percentage points from 67.1% (95% CI 65.0%-69.3%) to 73.7% (95% CI 72.5%-75.0%). Their probability of setting the correct diagnosis (nevus, melanoma, dermatofibroma, etc) increased by 10.5 percentage points from 42.5% (95% CI 40.2%-44.8%) to 53.0% (95% CI 51.3%-54.9%). Learning curves were most steep during the first 100 cases, as shown in Figure 6.

A descriptive analysis of the 39,022 diagnostic evaluations made by the participants during the entire training period showed that 72.6% of the malignant cases were correctly classified as malignant. Lesions were misclassified as malignant or benign 26.9% and 27.4% of the time, respectively. The most often misclassified benign lesions were seborrheic keratoses, compound nevi, and junctional nevi, which were classified as malignant in 31.9%, 28.9%, and 27.8% of the assessments, respectively. Lentigo maligna, nodular melanoma, and melanoma in situ were misclassified as benign in 48.7%, 42.2%, and 42.1% of the assessments, respectively. Participants guessed a different diagnosis than the correct diagnosis most commonly when assessing dermal nevi (misdiagnosed in 66.1% of cases), nodular melanoma (misdiagnosed in 63.4% of cases), and lentigo maligna (misdiagnosed in 61% of cases). See Table 3 for further details on the distribution of the participants’ answers on these and all other diagnoses.

During the 8-day intervention period, participants from the intervention group, on average, spent 2 hours 26 minutes (ranging from 0 minutes to 8 hours 35 minutes) practicing pattern recognition (quizzing) and 41 minutes (ranging from 0 minutes to 3 hours 23 minutes) reading the written educational modules included in the app. The majority of the participants’ reading activity was done at the beginning of the period and fell drastically after the first 100 cases (solid blue line in Figure 4). The average time spent diagnosing each case decreased from 36 to 28 seconds (solid red line in Figure 4).

The participants in both groups initially had identical relatively low confidence in their diagnostic ability to diagnose skin lesions using a dermoscope of 1.6 on a scale from 1 (low) to 4 (high), which increased significantly for participants in the intervention group by 32.9% to 2.1% (P<.001) after the intervention, but not for the control group (P=.23).

Participants who used the LIIR increased their ability to set the correct diagnosis by 10.5 percentage points and their ability to correctly classify lesions as benign or malignant by 6.6 percentage points. The steepest part of the learning curve was seen during exposure to the first 100 patient cases. This is consistent with previous studies [31,39,40]. Our findings align with previous research, indicating no direct relationship between years of experience and diagnostic competence in assessing skin lesions [2].

This study of 115 PCPs is one of the largest randomized controlled trials testing an educational intervention on PCPs’ ability to diagnose skin cancer [30] and the most extensive study using self-paced learning with a LIIR as the main component in the educational intervention. The participants were given no compensation for their time spent, which was done primarily in their spare time. Despite the trial period extending across the busy time before and during Christmas and New Year, the study had a high level of adherence. Of the 96 participants randomized to the intervention, 82 (85%) received it. A total of 12 participants did not manage to download and install the mobile app, which was yet to be released to the general public at the time. Despite this, 54 (62%) received the educational intervention as per protocol, and there were no dropouts from the control group. This level of adherence is similar to or higher than that of previous studies with a comparable intervention [30,41-43]. Post hoc calculations revealed a 98.1% statistical power for the intention-to-treat and 99.8% for the per-protocol analysis.

Limitations of this study include the recruitment method of inviting doctors at continuing education sessions about skin cancer. These doctors were perhaps more interested in or concerned about skin cancer diagnostics than the general population of PCPs. Evidence of this was that our participants had a relatively high initial diagnostic accuracy, most (84%) reported having access to a dermoscope in their clinic, and the percentage of participants who had received training (not including peer-to-peer and self-initiated training) in dermoscopy was also relatively high (46%). The effect of the educational intervention would possibly be more distinct in the general PCP population, where a lack of specific training in skin cancer diagnostics is more common [22]. Our study did not test long-term retention, and it is likely that the acquired skills will fade without continued use of the educational material [29]. Another limitation was that the 2376 patient cases presented in the LIIR were extracted from a department of dermatology and therefore likely to be more difficult than what is generally seen by the PCP. Therefore, the diagnostic accuracy found in this study might not reflect the PCPs’ diagnostic accuracy on the patients they meet in their clinic. Another discrepancy between diagnosing cases using the LIIR and the clinical examinations of a patient’s skin is that when encountering an irregular nevus in the clinic, examination of the patient’s other nevi may reveal that the nevus resembles the patient’s other nevi and therefore not an “ugly duckling,” but rather “regularly irregular,” reducing the suspicion of melanoma. Yet, with these cases being evaluated 39,022 times, it does reveal the most common diagnostic pitfalls in the population and where to intensify future educational interventions: SK, early melanomas, and nodular melanoma.

It was our initial hypothesis that PCPs, who possess the ability to apply the educational material in a clinical context, would exhibit a steeper learning curve with a higher end point from using this educational tool compared to medical students [31]. Contrary to our hypothesis, we did not observe a difference in the learning curve between primary care providers and medical students. One potential reason for this could be that primary care providers have a greater awareness of the consequences of their diagnostic decisions, which may have limited their assessment of the patient cases. They may have reacted as in the clinical setting and rather chosen a more serious diagnosis than miss a potentially malignant lesion.

As PCPs are the first to triage patients with skin cancer, improving the diagnostic accuracy of PCPs is necessary as the incidence of skin cancer has been rising [44,45] and is expected to keep rising [46-48]. The underlying causes for this growing disease burden are likely multifactorial, including biological and demographic factors such as increased ultraviolet exposure and a rapidly aging population. Skin cancer screening and general awareness increase the number of patients referred and biopsied [49], straining an already hard-pressed health care system. Adding to that, it has been stipulated that revised histopathologic criteria and historic underdiagnosis [50] but also current overdiagnosis [49,51,52] may contribute significantly to the growing incidence of skin cancer. It has previously been shown that increased awareness of skin cancer of PCPs does not increase the number of patients biopsied or referred to the dermatologist, etc [53], but rather that education and use of a dermoscope reduce the number needed to biopsy to detect melanoma in the primary health care setting [54]. Our results show a marked increase in the participants’ diagnostic confidence, which often does not correlate with diagnostic competence. Yet, with PCPs feeling more confident in their abilities, they could engage in doing more digital sequential follow-up, which has higher diagnostic accuracy than single-appointment evaluations [25]. This could potentially reduce the number of referrals and biopsies even further, leading to fewer excisions and, thereby, fewer overdiagnosed “melanomas” [55,56].

With the results of this study, we can address one of the leading causes of reluctance toward using dermoscopy in general practice: The time needed for training [57]. As digital education is easily accessible, can be acquired flexibly and on-demand, and does not require the participant to travel to and from the educational institute, it might be a more efficient way of increasing the diagnostic accuracy of multiple participants over a short period at a low cost. Our results show that the time needed to improve diagnostic skills using dermoscopy might not be more than a few hours at one’s own pace.

The perspectives of these results are potentially quite important; however, this trial did not test whether the observed improvements in diagnostic accuracy transfer to the participants’ clinical diagnostic accuracy and, in turn, if it changed their clinical patient management. This transfer of knowledge and its effect on clinical patient management should be the focus of future research.

Using self-paced training in skin cancer diagnostics using a digital LIIR with patient cases delivered by a quiz-based mobile app improves PCPs’ diagnostic skills and confidence. The time spent by each participant does not need to be very long, nor must it be done in 1 sitting. Significant improvements can be seen from an average of 3.5 hours over the course of 8 days.