In this study, we compared the performance of CNNs trained on standardized and nonstandardized images when classifying skin cancer as malignant or benign on 3 separate external data sets.

This retrospective comparative study was approved by the Monash University Human Ethics Committee (Project ID 28130).

In all, 3 CNN models with the same architecture were trained on International Skin Imaging Collaboration (ISIC) 2019 [14-17] and MoleMap (MoleMap NZ Limited) [2] data sets. Model architecture used ImageNet pretrained ResNet-50 as a backbone (Figure 1) combined with a transformer [18,19]. The ResNet-50 backbone was incorporated because of the trade-off between accuracy and complexity. A transformer was also added to the model to overcome the limitation of CNN in the context of learning global images. The same 3 CNN models were then additionally trained with a ResNet-18 backbone on either the ISIC 2019 (CNN nonstandardized [CNN-NS]) or MoleMap (CNN standardized [CNN-S] and CNN standardized number 2 [CNN-S2]) data sets.

CNN-NS was trained on 25,331 nonstandardized ISIC dermoscopic images consisting of 8 skin conditions (Table 1). We define nonstandardized images as images that are taken using multiple image capture technologies (Figure 2). CNN-S was trained on 177,475 standardized, teledermatologist-verified, clinical, and dermoscopic MoleMap images. This data set includes a total of 65 skin conditions organized into a 3-level hierarchical semantic tree (Table 1). This model was trained on standardized images taken using the same camera (DermLite FOTO System). CNN-S2 was trained on 25,331 standardized, teledermatologist-verified, and dermoscopic MoleMap images consisting of 8 skin conditions (Table 1). CNN-NS and CNN-S2 were trained on the same number of images and skin conditions, only differing in the standardization of the images the models were trained on.

CNN performance was assessed using 3 separate test data sets that were not used in model training.

Statistical analysis was performed using Python software (version 3.8.13; Python Software Foundation) and Stata statistical software (version SE 17; StataCorp). The primary outcome measures were sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) for the binary classification of lesions.

For each input image, the CNNs provided a score between 0 and 1 representing the probability that the input image is malignant. In binary classifications, thresholds are applied to the CNN models to establish the point at which an input image is labeled malignant. This threshold is variable and allows for the manipulation of the sensitivity and specificity of the models.

The performance was assessed by aligning the sensitivity and specificity of the CNN models to the teledermatologists’ and by calculating the AUROC. AUROC allows for the direct comparison of different models regardless of the threshold applied. Delong nonparametric test was used to evaluate the statistical difference between AUROC values resulting from the same data set. Additionally, 95% CI for the AUROC was computed using 2000 stratified bootstrap replicates. McNemar test was used to compare the sensitivities and specificities of the CNN models. The 1-sample, 2-tailed t test was used to compare the mean sensitivities and specificities of the teledermatologists against the sensitivities and specificities of the CNN models. P values <.05 were considered to have statistically significant differences.

During training, each model was internally validated on their training images. The model trained on nonstandardized images (CNN-NS) showed an AUROC of 0.950, whereas both models trained on standardized images (CNN-S and CNN-S2) showed an AUROC of 0.960 and 0.877, respectively (Figure 3).

Each CNN model was tested on the externally sourced Danish test set of 569 images. CNN-NS performance fell with an AUROC of 0.759 (95% CI 0.714-0.802). CNN-S outperformed CNN-NS when examined on the Danish test set, with an AUROC of 0.861 (95% CI 0.828-0.894), showing significantly greater generalizability than CNN-NS (P=.001; Figure 3). CNN-S2, the standardized model trained on the same number of images as CNN-NS, also outperformed the model, showing an AUROC of 0.831 (95% CI 0.789-0.869; P=.009). Among the standardized models, CNN-S had the greatest AUROC (0.861 vs 0.831; P=.06).

When tested on the externally sourced UQ test set of 422 images, CNN-NS performed well with an AUROC of 0.850 (95% CI 0.812-0.887). CNN-S outperformed CNN-NS when tested on the UQ image set, with an AUROC of 0.876 (95% CI 0.842-0.911), again showing greater generalizability than CNN-NS (P=.08; Figure 4). CNN-S2 also achieved a slightly greater AUROC (0.864, 95% CI 0.828-0.900) compared to CNN-NS, though this was not statistically significant (P=.35). Among the standardized models, CNN-S had the greatest AUROC (0.8765 vs 0.8638), though the difference was not statistically significant (P=.23).

When tested on the publicly available ISIC 2020 test set of 33,126 images, the performance of CNN-NS was reduced, with an AUROC of 0.763 (95% CI 0.743-0.783). CNN-S significantly outperformed CNN-NS when examined on the ISIC test set (P<.001), with an AUROC of 0.828 (95% CI 0.812-0.843), showing greater generalizability than CNN-NS (Figure 5). CNN-S2 also significantly outperformed the CNN-NS (P<.001), with an AUROC of 0.815 (95% CI 0.799-0.830).

Teledermatologists (N=4) were split into 2 groups, teledermatologists 1 and teledermatologists 2. To evaluate the performance of the teledermatologists against the CNN models, we used the mean sensitivity and specificity of the 2 teledermatologist groups as a standard. On the Danish images, the teledermatologists achieved a mean sensitivity of 82.9% (95% CI 80.8%-85.0%) and specificity of 79.2% (95% CI 78.5%-79.9%).

The CNN models’ malignancy threshold score can be manipulated, which can change the sensitivity and specificity of the models. To compare the performance of the models to each other, we first matched the sensitivity to that of the teledermatologists (82.9%). CNN-S achieved a specificity of 72% (95% CI 66.9%-75.9%), outperforming both CNN-S2 (62%, 95% CI 55.7%-65.3%; P=.02) and CNN-NS (45%, 95% CI 38.4-49.6; P=.001). Additionally, CNN-S2 revealed a greater specificity than CNN-NS (P=.001). Next, we matched the specificity of each model to that of the teledermatologists (79.2%). CNN-S showed a sensitivity of 74.7% (95% CI 67.8%-81.8%), outperforming both CNN-S2 (71.5%; 95% CI 63.8%-78.4%; P=.77) and CNN-NS (56.3%; 95% CI 48.2%-64.2%; P=.006). Additionally, CNN-S2 revealed a greater sensitivity than CNN-NS (P=.003).

To compare models’ performance to that of the teledermatologists, we compared the mean sensitivity (82.9%) and specificity (79.2%) of the teledermatologists to that of each model. This comparison revealed that our highest performing model (CNN-S) had a sensitivity (74.7% vs 82.9%; P=.10) and specificity (72.0% vs 79.2%; P=.053) comparable to that of the teledermatologists (Table 3). However, both CNN-S2 and CNN-NS had significantly lower specificity and CNN-NS had significantly lower sensitivity when compared to the teledermatologists (Table 3).

When taking the image quality of test set 1 into consideration, the AUROCs of CNN-NS, CNN-S, and CNN-S2 increased as the quality of images improved (Figure 6). CNN-NS showed an AUROC of 0.591 (95% CI 0.389-0.778), 0.757 (95% CI 0.670-0.835), and 0.794 (95% CI 0.741-0.844) for images of poor, fair, and good quality, respectively. CNN-S showed AUROCs of 0.742 (95% CI 0.602-0.864; poor quality), 0.847 (95% CI 0.792-0.879; fair quality), and 0.886 (95% CI 0.817-0.909; good quality), and CNN-S2 showed AUROCs of 0.735 (95% CI 0.578-0.873; poor quality), 0.795 (95% CI 0.721-0.861; fair quality), and 0.864 (95% CI 0.820-0.909; good quality).

Our results provide evidence that models trained on standardized images outperform and, hence, achieve greater generalizability than models trained on nonstandardized images. In recent years, advances in machine learning have led to the development of models that can compete and even outperform dermatologists in the classification of skin cancer [7-11]. Although these models have been shown to perform well when tested on a subset of images from their training data set, the generalizability of these models to images taken in different clinical settings and with different devices is unknown.

The impact that the varying image acquisition devices and techniques have on CNN model performance in dermatology has not been explored in the literature to date; however, the lack of imaging standardization in dermatology has been highlighted. The collection, transfer, and storage of clinical and dermoscopic images are not standardized in dermatology and have implications on the creation of data sets for machine learning, the reproducibility of imaging, and accessibility to relevant metadata for the images [22,23].

The standardized models (CNN-S and CNN-S2) consistently outperformed the nonstandardized model (CNN-NS) on all test sets. The statistical significance was directly affected by the number of images in the 3 test sets, with fewer images in test set 2 resulting in a nonsignificant difference in performance. Larger test sets will have a more accurate measure of model performance, and this finding would need to be considered when reporting validation results.

The ISIC holds an annual challenge that invites contestants to create a model that is trained and tested on images provided by the ISIC. In the AI community, the model that wins the ISIC challenge often holds a reputation as one of the best available. However, if tested on external data, the same performance is not guaranteed. If models are both trained and tested on the same set of images, then they are subjected to overfitting and thus poorer generalizability. The quality of a model should therefore be judged on its performance on multiple external data sets from varying population groups.

Several studies have looked at the performance of CNN models compared to the performance of dermatologists. These models perform comparably and even outperform dermatologists when classifying skin cancers. However, it is important to note that the images used in test sets are often taken from the same data sets used in the training of the models [7-11]. It is important when comparing models to dermatologists that the CNN is externally validated. This validation provides a clearer indication of the performance of the models in comparison to dermatologists and their ability to generalize to external data sets.

In our study, when tested on test set 1, the teledermatologists outperformed all models. Interestingly, CNN-S was trained on Australian and New Zealand patients and generalized well to the Danish images. There was no statistical difference between the sensitivity and specificity of the teledermatologists and the matched sensitivity and specificity of CNN-S. It is important to note that the Danish teledermatologists were predominantly trained on Danish skin and had access to metadata and multiple image viewpoints for a single lesion, which the models did not have access to. Previous studies have shown that the addition of metadata and inclusion of both macroscopic and dermoscopic images of a lesion can improve the performance of the model [24,25]. Therefore, incorporating these features into future models will be important and may level the playing field when assessing performance against teledermatologists’ clinical assessment.

The Danish images used in our study were taken by general practitioners who were required to undertake training to use the image capture technology. However, there were some issues with the quality of the images: some lesions were not centered, several lesions may be present within a single image, and parts of lesions were not included within the image frame. As the image quality improved, the diagnostic performance of all models and teledermatologists also increased. This finding highlights the influence that image capture techniques and image quality can have on CNN models and teledermatologists’ diagnostic ability. This finding is also a consideration when designing models for integration into web-based tools or mobile apps with consumers as end users, as the quality of images taken by consumers on their smartphones will vary, especially in the absence of training.

Our study has several limitations. First, the MoleMap data set used to train our 2 standardized CNN models was labeled by dermatologists; however, only very few images were biopsy proven. Given that histopathology is the gold standard for diagnosis, some of these images may have been mislabeled, which could have an impact on the performance of the models. Second, in test set 1 with 569 images, we only had access to 221 biopsy-proven images. The remaining 348 images in the test set 1 were labeled by dermatologists, which allows for the possibility of mislabeling. Third, the quality of the images in the training data sets (ISIC and MoleMap) and the type of image modality may have played a part in the performance of the models rather than the standardization of the images. It is important to consider that the quality of the camera used in the standardized MoleMap data set is less variable than the nonstandardized ISIC 2019 data set, which may have led to a discrepancy in the performance. CNN-S was trained on a combination of dermoscopic and macroscopic images, whereas CNN-NS and CNN-S2 were trained only on dermoscopic images. This combination of image modalities may have had an influence on the strength of the CNN-S model. Additionally, the models are complex, making it difficult to understand the process behind their decision-making (ie, a black box). This is an important limitation of the models and of this study and will be addressed through the incorporation of explainable AI techniques in our future models. Finally, in test set 1, the number of lesions in each group becomes small when divided into images of poor, fair, and good quality. In future studies, it would be better to evaluate a larger data set split among the quality groups to more confidently assess the relationship between image quality and CNN performance.

In this study, CNN models trained on standardized images based on dermoscopic and macroscopic modalities performed better than a CNN model with the same architecture trained on nonstandardized images when tested on external image data sets. This finding has important implications for model generalizability in the binary classification of skin cancer. In test set 1, image quality also had a direct impact on the performance of the models. For future algorithm training, development, and registration, it is important that model generalizability is considered through the evaluation of model performance on external image data sets.