Efficacy of an Artificial Intelligence App (Aysa) in Dermatological Diagnosis: Cross-Sectional Analysis

Introduction

Background

Diagnostic and therapeutic decisions in dermatology are heavily influenced by the morphology of diverse skin lesions. Traditionally, dermatological diagnoses are established by integrating the patient’s medical history, clinical examination, and, in some instances, dermoscopic and histopathologic analyses [1]. As it is predominantly a morphological feature–dependent specialty, dermatology is a field best suited for incorporating artificial intelligence (AI) image detection and recognition capabilities for aided diagnosis [2-5].

Given the discrepancy in access to dermatologists around the world, it is extremely crucial to be able to address patients’ medical needs [6]. Less than 1 dermatologist is available for every 100,000 individuals in India, and the majority of these specialists work in urban areas [7,8]. The diversity of cutaneous disorders and their striking resemblance to each other make accurate and efficient diagnosis challenging for general physicians. A delayed diagnosis due to a lack of specialists might significantly impact the patient’s quality of life [9,10]. Moreover, the high frequency of complicated inflammatory skin illnesses and the rising incidence of skin cancer have contributed to a surge in demand for dermatologists that is anticipated to continue growing in the future. Considering the potential for future pandemics, the capacity to deliver high-quality care virtually will likely continue to play a significant role in medicine [6,11]. AI-driven image diagnosis may be the solution to resolving these issues, allowing general practitioners to accurately detect common dermatological disorders by feeding a clinical image to a smartphone app [7,12,13].

Several AI-based applications have been created to assist in interpreting clinical pictures for various skin disorders, which are available for general use. By using these applications to examine concerning lesions, users may be prompted to schedule a telemedicine consultation or visit a dermatologist in person [6]. Medical personnel should have a thorough understanding of the merits and limitations of AI to promote its safe and efficient implementation [3,14]. Some of its merits include automating redundant assignments, performing constrained tasks, addressing spectator dependability issues, and ability to think outside the box. Conversely, there are unresolved legal, ethical, privacy, and liability issues associated with AI, and the inability to understand the decision-making process (ie, the “blackbox” nature) may limit its acceptability [2].

Despite the abundance of AI-integrated health apps accessible to the general public, there is limited research on their reliability, precision, and safety [6,15,16].

The Aysa AI App

Aysa is an AI-enabled symptom-checker app developed by VisualDx. Aysa combines a problem-oriented clinical search with a well-curated medical image database comprising more than 120,000 medical images pertaining to 200 skin conditions in all Fitzpatrick skin types, expert medical knowledge, and cutting-edge machine learning (ML) techniques. The app uses the in-device framework such as Apple’s CoreML in iOS to accelerate ML tasks. Aysa can modify its results based on a user’s medical history, further personalizing the experience for consumers. The Aysa app is commercially available for download on iOS and Android devices [17].

By analyzing clinical images, patient demographic details, skin type, the morphology of the lesions, and associated symptoms, the app provides probable diagnoses for skin conditions and gives a detailed overview of the condition along with the urgency of consultation. This enables the user to learn more about their skin issues and make informed decisions, although it is not intended for diagnostic purposes. Image recognition and analysis occur on the device itself using the in-device AI framework. However, there is a lack of information regarding the type of neural network the app uses. Privacy is ensured by encrypting images during transit, which are then discarded after analysis. Patient profiles, associated cases, and images are in complete control of the user [17].

Although the app is marketed as a symptom-checker app and not for diagnostic purposes, it is imperative to determine its accuracy and reliability, as the general public might be misled by the results.

Objective

The aim of this study was to validate an AI-based app (Aysa) as a preliminary diagnostic tool for Asian users with Fitzpatrick skin types III-V living in a semiurban town in India seeking consultation in a tertiary-care hospital for common skin conditions such as dermatitis, disorders of keratinization, papulosquamous disorders, pigmentary disorders, photodermatoses, skin infections and infestations, tumors, and other inflammatory conditions.

Methods

Source of Data

This observational cross-sectional study included 700 participants older than 2 years who consulted the dermatology outpatient department of a tertiary-care facility [Shri B M Patil Medical College, Hospital and Research Centre, BLDE (Deemed to be) University, Vijayapura, Karnataka, India] for common skin conditions between January 2023 and March 2023. All included patients were of Asian ethnicity with Fitzpatrick skin types III-V and presented with various skin conditions, which were grouped into the categories listed in Table 1. Malignant tumors were histopathologically confirmed. Hair and nail disorders and bullous disorders were excluded as the app is not designed to identify these conditions. Patients who had received prior treatment for their conditions and those who refused to authorize the inclusion of their images for the study were excluded.

Ethical Considerations

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Ethics Committee of BLDE (Deemed to be University; IEC/No. 09/2021). Informed consent was obtained from all individual participants and data were anonymized. No compensation was provided for study participation.

Methodology

This manuscript has been prepared following the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) checklist [18]. After detailed history and examination of the patients, the clinical diagnosis was established and verified by two expert dermatologists. Histopathological confirmation was obtained for suspicious lesions. Images of the skin lesions were captured on an iPhone 11 with a 12-megapixel sensor in a well-lit environment ensuring privacy. These images were then uploaded onto the Aysa app. A patient profile pertaining to age, sex, and skin type was created. Following this, the app identified the morphology of the skin lesions and ascertained the lesions by providing a description in colloquial language with pictorial representations. The location of the lesions was plotted on a human model put forward by the app, and certain questions relating to the duration of the skin lesions and associated symptoms were answered. Figure 1 provides images from the app depicting the workflow.

The app identifies 8 probable differential diagnoses for every skin condition. These were compared with the clinical diagnosis established by dermatologists.

‎

Statistical Analysis

Performance criteria such as sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, and F₁-score were used to assess the model’s performance. Disease-specific sensitivity; specificity; PPV; NPV; accuracy; F₁-score; and overall top-1, top-3, and all-8 sensitivities of the model were determined and represented as percentages with 95% CIs. The clinical diagnosis had to be predicted among the top one, top three, and all probable diagnoses to be eligible for top-1, top-3, and all-8 sensitivities, respectively. Data were analyzed using JMP Pro 16 software version 16 (SAS Institute). Categorical variables were compared with the χ² test and statistical significance was considered at P<.05.

Results

Demographics and Basic Characteristics

This study involved a total of 700 patients. More than half the sample comprised male patients (n=418, 59.7%) and the greatest proportion of patients were in the age range of 10-19 years (n=178, 25.4%). Patients presented with a wide range of conditions, which were grouped into 12 categories: bacterial infections (n=22, 3.1%), benign tumors (n=40, 5.7%), dermatitis (n=55, 7.8%), disorders of keratinization (n=28, 4.0%), fungal infections (n=97, 13.8%), malignant tumors (n=20, 2.8%), other inflammatory disorders (n=110, 15.7%), papulosquamous disorders (n=70, 10.0%), photodermatoses (n=21, 3.0%), pigmentary disorders (n=101, 14.4%), skin infestations (n=28, 4.0%), and viral infections (n=108, 15.4%).

Performance of the App

The AI model demonstrated an aggregate top-1 sensitivity of 71% (95% CI 61.5%-74.3%), top-3 sensitivity of 86.1% (95% CI 83.4%-88.6%), and all-8 sensitivity of 95.1% (95% CI 93.3%-96.6%). The top-1, top-3, and all-8 sensitivities; specificity; PPV; NPV; accuracy; and F₁-score of the grouped skin conditions are provided in Table 2. The top-1 sensitivities of skin infestations, disorders of keratinization, other inflammatory conditions, and bacterial infections were 85.7%, 85.7%, 82.7%, and 81.8%, respectively. All the classes displayed high specificity, accuracy, and NPV. All categories showed a significant association between clinical and probable top-1, top-3, and all-8 diagnoses (P<.001).

Table 3 shows the top-1, top-3, and all-8 sensitivities; specificity; PPV; NPV; accuracy; and F₁-score of the most common individual skin conditions found among the broader categories. The top-1 sensitivities of acne, dermatophytosis, psoriasis, lichen planus, and vitiligo were 93.2%, 72.2%, 81%, 27.7%, and 97%, respectively. The confusion matrix between probable top-1 diagnoses and clinical diagnoses is illustrated in Figure 2. Figure 3 depicts representative clinical images with their corresponding clinical and predicted diagnoses.

‎

‎

Discussion

Key Findings

This study analyzed the diagnostic accuracy of a commercially available AI-based health care app for various skin conditions. The app uses ML to analyze the clinical images, predict the probable diagnoses, and provide personalized guidance to the user.

Most of the patients included in this study had inflammatory conditions, pigmentary disorders, and infectious diseases. The top-1, top-3, and all-8 sensitivities for the AI model were collectively 71% (95% CI 61.5%-74.3%), 86.1% (95% CI 83.4%-88.6%), and 95.1% (95% CI 93.3%-96.6%), respectively. The app demonstrated high sensitivities in most categories in top-1 probable diagnoses, except in benign tumors, dermatitis, malignant tumors, and photodermatoses. When the top-3 probable diagnoses were considered, the sensitivities increased in all the categories except malignant disorders. In the case of photodermatoses, the sensitivity increased from 33.3% to 61.9% and subsequently to 100% when top-3 and all-8 probable diagnoses were considered, respectively. However, in the case of malignant disorders, the sensitivity remained the same and only increased to 25% when all 8 probable diagnoses were taken into account.

When considering specific skin conditions, the app could diagnose acne, dermatophytosis, psoriasis, and vitiligo with good sensitivity. Among papulosquamous disorders, the top-1 sensitivities of psoriasis and lichen planus were 81% and 27.7%, respectively. Among other inflammatory disorders, the top-1 sensitivity of acne was 93.2%, which increased to 100% when top-3 diagnoses were included.

Examination of the confusion matrix showed that the number of false negatives for herpes zoster was equal to the number of true positives, with herpes simplex being the most predicted diagnosis among false negatives (predicted in 43.7% of all patients with herpes zoster). This can likely be attributed to the morphology and location of the lesions. Most basal cell carcinoma cases (76.9%) were predicted as melanoma in the top-1 diagnosis.

Comparison With Similar Studies

We further sought to compare the diagnostic accuracy of the Aysa app with similar algorithms under comparable study conditions. However, direct comparison would only be possible if the same image sets were used in the evaluation of various algorithms.

Marri et al [2] assessed the Tibot AI app in diagnosing skin conditions in 600 patients. For the predicted top-3 diagnoses given by the app, the mean prediction accuracy was 96.1% (95% CI 94.3%-97.5%) and for the exact diagnosis it was 80.6% (95% CI 77.2%-83.7%).

Using clinical photos of skin lesions from patients with verified COVID-19, healthy individuals, and 18 common dermatoses, Mathur et al [19] developed a convolutional neural network (CNN)–based algorithm. The top-1 overall sensitivity for the diagnosis of 20 skin disorders was 87.65%, while the top-3 sensitivity was 96.72%.

Table 4 provides a comparison of the sensitivities, specificity, and PPV of AI algorithms of this study and the studies by Marri et al [2] and Mathur et al [19] in diagnosing various skin disorders. The sensitivity in the majority of the conditions was comparable in all the studies except for lichen planus and malignant tumors. Although the Tibot app evaluated by Marri et al [2] demonstrated higher sensitivity in diagnosing malignant tumors, it only gives a broad diagnosis, unlike the Aysa app, which predicts a specific diagnosis. In the study by Mathur et al [19], the CNN model predicted lichen planus with better sensitivity than achieved with the Aysa app.

Wu et al [20] evaluated the accuracy of a CNN model in diagnosing inflammatory skin conditions. The sensitivity and specificity of the model were found to be 94.4% and 97.2%, respectively, and the overall accuracy was 95.8%. For eczema and atopic dermatitis, the accuracy was 92.57%, with a sensitivity and specificity of 94.56% and 94.4%, respectively. The accuracy for psoriasis was 89.46%, with a sensitivity and specificity of 91.4% and 95.48%, respectively. In this study, the Aysa app showed an accuracy of 98.3% with a top-1 sensitivity of 81% and a specificity of 99.8% in the case of psoriasis. For atopic dermatitis, the accuracy was 91%, with a top-1 sensitivity and specificity of 52.7% and 94.3%, respectively.

Other studies have demonstrated the efficacy of AI in diagnosing benign and malignant dermatoses [21-23]. The performance of the CNN models evaluated by Esteva et al [22] and Han et al [23] was comparable to or better than the diagnostic ability of dermatologists. In this study, the Aysa app demonstrated a top-1 sensitivity of 62.5% and a specificity of 99.5% in identifying benign tumors. For malignant conditions, the top-1 sensitivity was 10% with a specificity of 99.8%.

Implications

The Aysa app has proven to be effective in predicting most of the common dermatoses encountered in a population. In addition to skin analysis, the app provides in-depth details on the conditions in the form of an overview comprising the causes, symptoms, risk factors, course, prognosis, and treatment information; preconsultation advice; when to see a doctor; and differential diagnoses. Materials adapted from renowned textbooks, journal papers, PubMed, the World Health Organization, the Infectious Diseases Society of America, and the US Centers for Disease Control and Prevention are included in the content [17]. Notifying the patient of the urgency index is practical because skin disorders are typically ignored until they cause significant inconvenience. Thus, the Aysa app has the potential to motivate patients to seek medical care, improve patient engagement and participation, improve the efficiency and productivity of physicians, and reduce health care expenditure [24,25].

Health care practices can be enhanced by integrating advanced diagnostic knowledge using these AI-based health care systems. For a skin condition, images can be uploaded to a specialized dermatological AI system from a general practitioner’s office, and prompt analysis can be performed if the uploaded image is sufficient to reach a conclusion. This would help patients with low-risk conditions receive immediate reassurance about their concerns, while those with high-risk conditions can have a speedy referral to a specialist clinic [12]. Finding a balance that optimizes the advantages of AI while maintaining the humanistic touch is crucial for patient care.

Limitations

Absence of image consistency in terms of focus, angle, and illumination is one of the main limitations of our study. Although the app can identify almost 200 skin disorders, this study included only 46 common conditions. The majority of the study population had infections, pigmentary disorders, and inflammatory illnesses. Photodermatoses and tumors were relatively less frequent in this population, which may account for the app’s poor performance in these categories. Additional research focusing on these conditions and others not included in this study may be required to validate the app’s performance. For simplicity of comprehension, specific skin conditions were categorized into broad groups. This could have given an impression of relatively consistent performance, as in the case of papulosquamous disorders, where the app showed good sensitivity to diagnose psoriasis but failed to diagnose lichen planus with the same sensitivity. Dermatological conditions have a diverse morphology based on various factors, including severity of the disease. This might hinder the ability of the app to provide an accurate diagnosis. Further studies correlating severity of the disease and other factors with the app’s diagnostic ability might be required.

There are certain drawbacks to the app. As it is designed for users above the age of 2 years, certain conditions such as infantile hemangioma, commonly encountered in clinical practice, could not be diagnosed. As the app is intended for assessing skin conditions, hair and nail disorders could not be included in the study. The preconsultation advice provided by the app contains information regarding over-the-counter medications appropriate for the condition. This may encourage the patient to self-medicate rather than seek consultation. Another limitation is the lack of transparency regarding the type of neural network used by the app despite our efforts to obtain that information.

Conclusions

The Aysa app has demonstrated promising outcomes in the diagnosis of prevalent dermatological issues such as infections, inflammatory disorders, infestations, and pigmentary disorders. However, the app is unreliable at detecting photodermatoses and malignant tumors. Further improvement might be required for the app to be implemented in clinical practice.