The Store-and-Forward Telemedicine Service User-satisfaction Questionnaire (SAF-TSUQ) [28], a Dutch validated questionnaire, consists of 29 closed-ended questions within the following themes: Training (eg, offered training to work with teledermatology), Support Communication (eg, preferred way to communicate with Ksyos), Interaction (eg, teledermatology is easy to use), and Use (eg, teledermatology improves access to dermatology care) with the telemedicine platform (Multimedia Appendix 1). The responses are rated on a 5-point Likert scale (1=strongly disagree to 5=strongly agree) with 2 additional options “I do not know” and “not applicable.” On the basis of this questionnaire, the teledermatology service of Ksyos was evaluated [29]. Ksyos is a digital hospital and the largest teledermatology provider in the Netherlands [29].

We added 21 closed- and 6 open-ended questions to tailor the generic SAF-TSUQ questionnaire to specifically address the nuances of teledermatology use. These additional questions were added for the following themes: demographic information of dermatologists, the use of teledermatology (during the COVID-19 pandemic, eg, experience and frequency of use before or during the COVID-19 pandemic), the use of teledermatology by GPs (eg, photo quality), and to evaluate teledermatology as experienced by the dermatologists (eg, not able to perform a full body skin check; Multimedia Appendix 1). All questions were mandatory, except the optional free text fields at the end of each theme. These new questions were discussed in a focus group with teledermatology experts of Ksyos and reviewed by a dermatologist via email.

The supplemented questionnaire was sent by email to dermatologists who (1) had an active teledermatology platform account at the moment of sending the questionnaire, and (2) responded to a teledermatology consultation between October 1, 2019, and September 30, 2021.

The questionnaire was sent out in December 2021. Dermatologists gained access to the questionnaire via a unique link in the email. The link became inactive after the questionnaire was filled in so the questionnaire could only be filled in once. This link only allowed tracking if a questionnaire was filled in, thus the individual responses were not visible for the researchers. We sent reminders once to nonresponders 1 week after sending the questionnaire. Informed consent was requested in the questionnaire for the use of their responses for research purposes.

Participation was voluntary and anonymous. Four gift cards were raffled among respondents who completed the questionnaire in a larger study. The LimeSurvey questionnaire tool was used to store the questionnaire and its responses. The Medical Ethical Committee of the Amsterdam University Medical Center (location AMC) provided a waiver.

The responses to the closed-ended questions of the SAF-TSUQ questionnaire were analyzed descriptively per questionnaire theme with RStudio.

The Sittig and Singh sociotechnical model was used for the analysis of the open-ended questions [30]. This sociotechnical model has been previously used to understand sociotechnical barriers and facilitators of digital health use after its implementation [31-34], and consists of eight interrelated dimensions: (1) hardware and software; (2) clinical content; (3) human-computer interface; (4) people; (5) workflow and communication; (6) internal organizational policies, procedures and culture; (7) external rules, regulations, and pressures; and (8) system measurement and monitoring (Table 1).

The open-ended questionnaire responses were analyzed by combining an inductive and a deductive approach. First, during a content analysis with an inductive approach, 2 researchers (LWP, Bibiche Groenhuijzen) systematically and independently reviewed the open-ended questionnaire responses to identify patterns and themes using a grounded theory approach [35]. Secondly, a deductive approach was applied to map the identified themes onto the sociotechnical model. The combined deductive and inductive approach allowed for the potential extension of the Sittig and Singh [30] model.

During the content analysis, the data were ordered into discrete responses during the open coding phase. Responses consisting of multiple sentences were split into smaller units of meaning, if necessary. An inductive coding approach was applied by one researcher (LWP) and one coder (Bibiche Groenhuijzen), meaning that the first codes emerged from the prior content analysis. This is an iterative approach whereby the data are reread and the first codes are refined and, if required, new codes can be created. These refined and new codes were (re)applied to the data until saturation of codes was reached and all responses were coded. This final set of codes is referred to as our set of “subcodes.” The subcode “not applicable” was assigned to responses that could not be interpreted by the researchers, and those responses were, therefore, assigned with the maincode “not able to code” and were not assigned to a dimension. The subcodes represented the rich details of the responses. Patterns emerged from these subcodes from which maincodes were formulated. The maincodes represented higher-level categories of the data compared to the subcodes. During axial coding [36,37], sets of subcodes were grouped by one maincode. The open and inductive coding phase resulted in a codebook of sub- and maincodes (Tables S1 and S2 in Multimedia Appendix 2). The coders (Bibiche Groenhuijzen, FvS) independently assigned sub- and maincodes and were blinded to each other’s codes. Continuous discussion meetings took place between the two coders (BG, FvS) and a researcher (LWP) to discuss the content of the responses and to refine the sub- and maincodes. In addition, the dimensions of the sociotechnical model were discussed. Small adaptations to the definition of the dimensions were made to fit them to the teledermatology evaluation. This was done during the inductive phase of coding [30], since at this moment, we got insights into the meaning and application of the dimensions (Table 1).

Finally with a deductive approach, 1 of the 8 interrelated dimensions of the sociotechnical model [30] was independently assigned by the 2 coders to a response. There was a discussion meeting (Bibiche Groenhuijzen, FvS) until consensus was reached regarding the assigned sub- and maincodes and dimensions. A third researcher (LWP) was involved in the discussion meeting if no consensus was reached.

As part of our comprehensive evaluation, we will discuss teledermatology as a complex adaptive system (CAS) [26,27], characterized by several components that interact with each other as part of achieving broader system outcomes. Tenets of a CAS include emergent behavior (system interactions may result in an outcome that is unpredictable) and nonlinearity (a small change in one part of the system may lead to a large change in another part of the system) [38,39]. The components must be meaningfully integrated to achieve desired outcomes such as accurate diagnostic outcomes, improved patient care, and user satisfaction [9,40]. Teledermatology exhibits characteristics of a CAS where users (eg, patients, GPs, dermatologists) are separated by system concepts (eg, time and space), and different technologies (eg, digital systems and photography equipment). A CAS-guided evaluation allows us to understand how various system subcomponents interact within system operations, supplementing the sociotechnical approach by elucidating the relationship between multiple interacting components in a system.

After the questionnaire study, there was an open, unstructured panel discussion with 3 dermatologists (10, 12, and 13 years’ experience with Ksyos teledermatology platform) where one researcher (FvS) presented the questionnaire outcomes. The interpretation and perspectives of the dermatologists on the questionnaire results, and their experience with teledermatology was discussed. The remarks of the dermatologists were noted and directly mapped onto the adapted dimensions (Table 1).

The Medical Ethical Commission of the Amsterdam University Medical Center granted a waiver stating that the study did not require additional approval.

A total of 25 dermatologists (10%) completed the questionnaire, including informed consent (Table 2). We received 3 hard bouncers, 6 out-of-office emails (for an extended period of time), and 2 dermatologists who indicated that they did not want to receive the questionnaire and reminders. The median time to fill in the questionnaire was 11.50 (average 16.49) minutes.

Overall, dermatologists found that the training, support communication, and information provided by Ksyos was sufficient to work with the platform. However, they were more critical that teledermatology care is not considered the same as providing physical care (Figure 1).

Dermatologists were satisfied with performing the teledermatology tasks. They knew how to contact Ksyos, and if support communication with Ksyos was needed, they preferred to do this via email.

While they found that performing teledermatology did not help them to increase their dermatology knowledge, dermatologists expressed that the platform is easy to use and understandable. Despite this positive feedback, one-third of the dermatologists mentioned that the platform did yet not contain all functionalities to provide teledermatology care (eg, they expressed their preference to provide a diagnosis in a free text field).

The majority of dermatologists acknowledged that teledermatology serves as a valuable medium to provide the GP with advice, recommended the platform and stated that they would use it again.

Some dermatologists reported that teledermatology use slightly increased during the first COVID- 19 wave, compared to prepandemic levels (Table 3). During this time, they noted that sufficient support from Ksyos on how to use teledermatology was received. Dermatologists observed that the type of questions asked by GPs in the teledermatology consultation did not change. They reported difficulties in evaluating teledermatology consultations with poor photo quality and anamnesis information.

A codebook consisting of 13 maincodes and 113 subcodes was developed (Tables S1 and S2 in Multimedia Appendix 2). Two coders (FvS, Bibiche Groenhuijzen) had a 65.5% and 71.4% agreement after coding the responses with the sub- and maincodes, respectively. Out of the 247 responses that were coded, the 2 coders were not able to code 35 responses (Table 4). We were able to map all responses on an existing dimension of the Sittig and Sing model (Multimedia Appendix 3).

Most dermatologists had a positive experience with teledermatology, especially during the COVID-19 pandemic where they could continue to provide care without seeing the patient at the hospital (Table 4. Q1).

Dermatologists (56%) noted that GPs generally provided complete information in the teledermatology consultations with the required types of photos and anamnesis. However, they were more critical regarding the use of teledermoscopy for diagnosing potential malignant skin lesions. The inability to perform a full body skin check of the patient is a barrier (Table 4. Q2). This underscores the need to receive (overview) pictures of the skin lesion, along with the patient anamnesis information. Dermatologists emphasized the importance of accurately selecting lesions for evaluation, given the ability of GPs to perform a full-body skin check (quality of care) (Table 4; Q3, Q4).

Dermatologists (44%) expressed their concerns about the low quality of the photos taken by the GP; these are often blurry and assessment (photo quality). Also, overview and dermoscopic photos are mostly lacking (quality of care), hindering the assessment of the skin type and number of lesions. Consequently, most dermatologists (76%) advise physical referrals (Table 4; Q5).

The accurate assessment of potential malignant skin lesions relies heavily on high-quality (dermoscopic) photos [41]. Seven dermatologists considered that teledermoscopy is unsuitable for this purpose, and prefer live dermoscopy for a clearer view of the skin (system use). Twelve dermatologists find teledermoscopy suitable and 5 dermatologists preferred not to use teledermoscopy, (Table 4; Q6) indicating different viewpoints for diagnostic suitability for different lesions (system use).

Several dermatologists indicate the importance of training GPs in providing high-quality photos and complete patient information (training, Table 4; Q9). They believe such training will help GPs understand which information is required for accurate assessment by dermatologists. Dermatologists assert that they feel adequately trained in using teledermatology due to provided instructions, and emphasize that teledermatology does not enhance their dermatological knowledge but assists GPs in diagnosing skin lesions (Table 4; Q7, Q8).

Three dermatologists find teledermatology more time-consuming than a physical consultation (workload), especially with poor-quality photos (quality of care). For these reasons, dermatologists prefer a physical consultation as they cannot replace the full body skin check (quality of care). Dermatologists believe that teledermatology is, therefore, only useful for dermatologists who have sufficient time. Other dermatologists indicate that teledermatology is indispensable, for example, to monitor patients with a known diagnosis and to supervise the GP.

Dermatologists differ in their viewpoint about the usability of the platform. While some believe that improvements have been made in the last years (system use), others mention that there are too many fixed, mandatory fields in the teledermatology consultation form which decreases their possibility to enter free text (System failure/improvement).

Dermatologists provide suggestions for system improvements, for example, that the photography equipment of the GP should automatically focus to yield better photos, or a mobile app to respond to teledermatology consultations (System failure/improvement).

Three dermatologists observed that GPs used teledermatology more often during the COVID-19 pandemic than before, driven by the urgency to be able to provide dermatological care.

Comments of the dermatologists mostly matched the questionnaire responses (Table 5). We obtained additional insight into the experiences of dermatologists with teledermatology, since also new findings compared to the questionnaire responses were mentioned. Dermatologists gave possible explanations for critical responses on the use of teledermoscopy provided by dermatologists in the questionnaire (Organizational policies and procedures, External rules, regulations, and pressures). There were no comments made regarding the dimension “System measurement and monitoring.”

Dermatologists showed overall satisfaction with the teledermatology service, and considered it a valuable addition to dermatology care, rather than a replacement of face-to-face consultations. Dermatologists find supervising GPs in patient and lesion management especially valuable. Teledermatology facilitates efficient triage, aiding in prioritizing appointments in secondary care, and thus, streamlines the referral assessment process for dermatologists, ultimately saving time in dermatology practice. Despite reported preferences for face-to-face consultations, dermatologists consistently reported high satisfaction rates with teledermatology [3,42,43]. Teledermatology provides appropriate low-complex diagnostic care at the right place, although acknowledging limitations. The current health care landscape is marked by challenges, including rising health care costs and long hospital waiting times. Teledermatology emerges as a solution to enhance efficient dermatology care [44]. However, barriers were also identified through qualitative analysis of the open-ended responses. Further, this discussion also explores factors that intricately influenced the dermatologists’ experience and satisfaction with the use of teledermatology.

First, dermatologists reported that incomplete anamnesis information or low-quality (dermoscopic) photos (eg, blurry, bad lightning), from GPs hinders an accurate diagnosis, as they heavily rely on this, due to the inability to conduct a full body skin check (System measurement and monitoring [45,46]. From the study’s holistic perspective, the trade-off between the convenience of teledermatology (saving patient travel time and less unnecessary physical referrals) and the dermatologists’ preference for physical consultations (depending on lesion complexity) is influenced by the quality of the provided clinical content.

The suboptimal quality of photos and anamnesis information may lead to more time-consuming teledermatology assessments (Workflow and communication), emphasizing the need for adequate clinical content quality. However, this illustrates a trade-off in teledermatology use, namely the balance between the photo quality and the time a GP spent on taking photos. High-quality photos will likely increase the workload of the GP, but will save dermatologists’ time during assessment and improve the dermatologists’ confidence and diagnostic performance [47,48]. In contrast, a consultation with low quality images will likely be considered ineligible for assessment by the dermatologists [3,49]. These barriers were acknowledged during the panel discussion. The poor image quality barrier was already identified in 2011 in the United States [9], making it remarkable that this continues to exist in 2020 in Denmark [4] and in our study. Despite the time-efficiency of teledermatology [3], this study did not assess how quality of the clinical content affects the time needed for assessment by dermatologists.

Teledermatology was already widely implemented in Dutch general practice prior to the pandemic, leading to a more gradual uptake compared to other countries [20]. Dermatologists did not observe a massive increase in the number of teledermatology consultations during or after the pandemic, which indicates its well-established use in the Netherlands. Despite the global forced uptake of teledermatology during the COVID-19 pandemic, both patients and their treating practitioners viewed the experience positively [50].

Our findings demonstrated how barriers impacted different aspects of care delivery. Exclusively analyzing the closed-ended questions of the SAF-TSUQ questionnaire would not have unveiled these findings, as those specifically focused only on the platform usage and omitted other aspects of teledermatology (eg, workload, clinical content). Applying a user-centered approach by surveying the dermatologists gave a better understanding of the dermatologists experiences and satisfaction with using teledermatology. This emphasizes the need for improvements to ensure that teledermatology remains of added value, facilitating its acceptance and implementation in general health care, rather than viewing it solely as a separate digital system [51].

From a sociotechnical CAS perspective, we aimed to understand the impact of interacting system components (eg, photo quality vs workload). While acknowledging that this study is limited in identifying all complex relationships of teledermatology use and that poor photo quality is not a new result, we provide a different perspective on how to think about those barriers and how they present trade-offs in managing these issues. The interrelated nature of digital health implementation means that we cannot view barriers as isolated entities but rather we must think of them as an interrelated system [30]. Our integrated sociotechnical-CAS perspective can be used to study digital health in other settings.

Future research could address the refinement of training methods for GPs to optimize their dermatology knowledge and use of the advanced photography equipment (People). The training should be time-efficient and flexible, given the high workload of GPs. We suggest that accreditation would be a nice incentive for participation. Second, automatic focus of camera equipment (Hardware and software), and artificial intelligence assisted (image) analysis tools to improve the quality of the provided clinical content should be investigated. The critical attitude of dermatologists toward teledermatology should be investigated. Despite advancements of photography equipment over the past decade and the widespread use of teledermatology, dermatologists remain critical. However, caution is needed when making aforementioned changes in teledermatology systems, since the barriers resulting from this study cannot be evaluated as an isolated entity. Changes could result in unintended consequences on various parts of the system. This underscores the complexity of teledermatology innovation. By looking through a complex system perspective, we could not only identify the barriers but also support the investigation of the consequential impacts on various aspects when modifying one simple factor to a CAS [26,27].

We achieved an acceptable response rate of 10%, considering the fact that the questionnaire was distributed during a lockdown. A comparable response rate (13%) was achieved in the study of Kennedy et al [52]. The panel discussion enriched perspectives, thereby balancing diverse viewpoints. This study is part of a larger investigation into teledermatology, wherein also the experiences of GPs were investigated [53]. Through the additional open-ended questions and panel discussion, we were able to provide in-depth insight into the responses of closed-ended questions of the SAF-TSUQ questionnaire. We thus believe that we obtained valid responses. This study was the first step to investigate the experiences of dermatologists with the use of teledermatology in the Netherlands. Follow-up research among a larger group of dermatologists is needed to enrich our results.

Secondly, we used a sociotechnical model as a theoretical framework to thematically analyze the responses on the questionnaire, ensuring a consistency and reliability across the data evaluation process. Through structured coding, we were able to systematically explore the responses, revealing interconnected themes such as that “workload,” which could be related to “Clinical content” (poor photo quality), “Workflow and communication” (time-consuming), and “System measurement and monitoring” (unable to conduct a full body skin check). More information could be gathered from research in practice in addition to the questionnaire responses (eg, observing and interviewing how dermatologists and GPs use the teledermatology service). While various frameworks exist, such as Rogers’ diffusion of innovations theory [54] and Reason’s Swiss Cheese Model [55], they may have a limited scope for the evaluation of complex systems. In this study, we chose the sociotechnical model of Sittig and Singh [30] that integrates technological and sociotechnical (eg, people, processes) factors. This sociotechnical model helped us to examine the experiences of dermatologists with teledermatology services in a systematic manner [56]. Given the fact that this teledermatology service is implemented since 1995 [5], this sociotechnical model accounts for long-term perspectives, allowing for a thorough analysis of the open responses per dimensions of the model.

We experienced the expected limitations of a questionnaire study. Some incomplete responses were excluded. Since this study was the first step in understanding the dermatologists’ experiences with teledermatology, we believe that a questionnaire combined with a panel discussion was an appropriate method. Follow-up research among a larger group of dermatologists within focus groups could provide a deeper understanding of the experiences. Using a sociotechnical model supported a consistent approach, but the sentiment of responses might be lost [57,58]. However, steps were taken to mitigate this through blinded coding and researcher discussions (FvS, LWP, and BG).

Another limitation included the focus on a single teledermatology service which may restrict the generalizability of our findings. It was unknown if the dermatologists also had experiences with other teledermatology services, and avoided this in the questionnaire for clarity purposes. It should be addressed in follow-up research if the experiences of Dutch dermatologists with the Ksyos teledermatology service are shared across different teledermatology services. This could be achieved by investigating a broader range of teledermatology services across different countries through surveying dermatologists or conducting panel discussions, with a similar methodology used in our study. We acknowledge that the teledermatology system in the unique Dutch health care setting, including our results, may not be generalizable to other health care systems. Additionally, only experienced dermatologists responded to the questionnaire, thus a selection bias may exist. Including dermatologists who use teledermatology less frequent in follow-up research could reveal broader perceptions. Finally, the questionnaire did not address reimbursement and legal aspects, which are important considerations for teledermatology adoption [9,10,56].

Our study sheds light on the use of teledermatology in the Dutch health care system. Dermatologists generally view teledermatology as a valuable tool to provide access to dermatology care as alternative or decision-making prior to a physical consultation. However, their feedback highlights its challenges in providing effective and convenient care via teledermatology. Currently, trade-offs exist between the convenience of saving patient travel time and the preference for a physical consultation, and the need for high-quality photos versus the time spent by GPs on taking pictures. Improving the photo quality and the completeness of the anamnesis information is crucial. Innovative solutions such as artificial intelligence–assisted analysis tools, and not only GP training, are deemed necessary. Complete teledermatology consultations are a prerequisite for dermatologists to provide added value to the GPs and patients. Addressing barriers and implementing solutions will facilitate the continuous use of teledermatology; however, we emphasize the need to address this from a complexity perspective to understand trade-offs as a means of preventing unintended consequences.