Evaluating artificial intelligence’s role in lung nodule diagnostics: A survey of radiologists in two pilot tertiary hospitals in China

Survey development

The radiology departments at Beijing Anzhen Hospital (Anzhen) and Beijing Tsinghua Changgung Hospital (Changgung) jointly administered an online survey titled “ Survey on Radiologists” Experience with AI-Assisted Lung Nodule Diagnosis Systems’ [Supplementary Figure 1]. Participants in the study comprised all radiologists affiliated with Beijing Anzhen Hospital and Tsinghua Changgung Hospital. The survey participants included radiologists of varying levels, ranging from residents to professors, and encompassing all levels of seniority at two hospitals. The research sample is the most comprehensive and complete representation of radiologists in tertiary hospitals located in developed cities in China. The questionnaire consisted of 26 questions, and respondents were estimated to require approximately 5 min to complete it. The Institutional Review Board approval was not deemed necessary for this study. This study is conducted on an anonymous basis, with all participants having provided signed informed consent forms.

The survey was structured into three distinct subparts. The initial segment comprised eight questions focused on respondents’ demographic information, including age, gender, affiliated hospital, years of professional experience, professional title, years of experience with AI, AI system-assisted subspecialty diagnosis, and AI-practiced techniques [Table 1]. Notably, no personal identifying data were collected in this section. The second part of the survey utilized a system usability scale (SUS) questionnaire consisting of ten questions designed to assess the usability of the AI system.^[14] The SUS is a widely recognized instrument employed in studies evaluating system usability. It features a combination of positively framed statements for odd-numbered items and negatively framed statements for even-numbered items. The third part consisted of eight questions probing participants’ sentiments and predictions regarding the integration of AI applications into radiological practice, both at present and in the forthcoming 5–10 years. The questionnaire, encompassing the second and third sections, comprised a total of 18 questions, employing a five-point Likert scale ranging from “completely disagree” (score: 1) to “completely agree” (score: 5).

AI-aided diagnostic system

At Beijing Anzhen Hospital, the Care.ai system,^[15] developed by Yitu Technology, was implemented in January 2019. This system possesses the capacity to autonomously detect and recognize lung nodules, providing detailed annotations encompassing size, location, density, and imaging characteristics of identified nodules, ultimately generating comprehensive diagnostic reports.

Meanwhile, Beijing Tsinghua Changgung Hospital adopted the Dr.Wise system in June 2021.^[16] Developed by Deep-wise Inc., the Dr. Wise system operates as a computer-aided detection and diagnosis system. It is engineered to autonomously and precisely detect and segment both solid and ground-glass lung nodules, subsequently conducting automated calculations for parameters such as nodule diameter, density, volume, mass, volume doubling time, and mass doubling time.^[16]

Statistical analysis

Quantitative variables were represented in the form of means, accompanied by standard deviations (SDs), and their respective ranges. Questions of categorical variables were expressed as proportions and percentages in relation to the total number of responses. Selected variables including gender, age range, geographical location, years of professional experience, professional title, and years of experience with AI were predetermined and included in subgroup analyses. Questions related to “AI practiced techniques besides lung nodule” and “AI system-assisted subspecialty diagnosis besides lung nodule” were designed as multiple-choice questions and analyzed through ranking methods. The cumulative SUS score obtained from the second part of the questionnaire was scored and converted to a scale of 0–100.^[17] Statistical analyses of the results were carried out using RStudio (Version R 4.1.2), and graphical representations were generated using Excel 2019 software. P < 0.05 was considered statistically significant.

Participant demographics

The demographic characteristics of the participants are presented in Table 2. A total of 90 valid responses were collected, comprising 57.78% female and 42.22% male participants. The largest cohort (37.78%) fell within the 30–39 age bracket, closely followed by the 18–29 age group (28.89%) and the 40–49 age group (21.11%). Regarding the workplace, 26 participants (28.89%) were affiliated with Beijing Tsinghua Changgung Hospital, while 64 participants (71.11%) were associated with Beijing Anzhen Hospital. Notably, nearly half of the respondents possessed over a decade of experience in this field. Moreover, a substantial portion held the professional title of “resident physician.”

Table 3 displays the results from utilizing the AI-assisted diagnostic system. As for experience utilizing AI systems, there was notable diversity, with the highest percentage of participants (22.22%) falling within the 1–2 years category.

Alongside lung nodule screening, the most frequently utilized AI techniques among participants included CT (79, 87.87%), followed by X-ray (34, 37.78%), and MRI (13, 14.44%). Furthermore, the AI systems most commonly employed were those focused on cardiovascular applications (69, 76.67%), followed by breast-related AI systems (52, 57.78%), and lung-related AI systems (44, 48.89%).

A comprehensive evaluation of AI technology

Figure 2 illustrates the outcomes of the comprehensive evaluation of AI technology. Radiologists reported that the AI system had a positive impact on their work, with mean Likert scale scores indicating an improvement in work efficiency (4.6 ± 0.6 [SD], range: 2–5) and enhanced diagnostic accuracy (4.2 ± 0.8 [SD], range: 2–5).

Close

Figure 2:

A comprehensive evaluation of artificial intelligence technology Box plots and whisker plots show the distribution of responses using a Likert scale ranging from 1 to 5. The vertical bar represents the median value; the 10^th and 90^th percentiles are set as whiskers limits and the 25^th and 75^th as box limits; the mean value is plotted as a “•.”

Export to PPT

Radiologists expressed their preferences regarding the allocation of responsibility for diagnostic errors when AI systems are in use, with an average Likert scale score of 3.7 ± 1.4 (SD), indicating a preference that the responsibility should primarily rest with the radiologist (range: 1–5). In addition, they conveyed a perception that they lacked sufficient training in imaging AI, as evidenced by an average Likert scale score of 3.6 ± 1.1 (SD) (range: 1–5). Radiologists held varying opinions on the potential impact of AI systems on the employment opportunities of imaging physicians in the next 5–10 years, with a tendency toward a mild agreement (average Likert scale score: 3.2 ± 1.4 [SD], range: 1–5). Moreover, radiologists exhibited a tendency to mildly disagree with the notion that medical imaging AI technology would ultimately replace radiologists in the long run (average Likert scale score: 2.5 ± 1.1 [SD], range: 1–5).

Radiologists unanimously acknowledged the potential positive impact of medical imaging AI technology on the imaging department over the next 5 years, with a mean Likert scale score of 4.2 ± 0.8 (SD) (range: 2–5). Furthermore, they demonstrated a strong inclination to endorse the use of AI systems to their peers, as reflected by an average Likert scale score of 4.4 ± 0.7 (SD) (range: 2–5).

Acceptable SUS scores

Our study delved into the assessment of system usability scores among radiologists concerning the AI-aided diagnostic system designed for lung nodules. This system operates by processing post-CT examination images, recognizing segmented images, and detecting nodules. The radiologists from both hospitals displayed a positive attitude toward the AI system as a whole. The usability scores were consistently rated as “good,” reflecting radiologists’ confidence in the system’s potential for frequent application and its ease of use. However, the perspectives on system consistency exhibited a degree of divergence. Furthermore, our study underscores the paramount importance of evaluating critical data characteristics, such as data quality and consistency, when integrating AI-assisted tools into clinical practice. These considerations are accompanied by a myriad of legal, ethical, and clinical acceptability issues that merit meticulous examination in the deployment of such technologies.^[20] The system-providing company is required to perform debugging and deployment tailored to the specific needs of each hospital, with continuous updates aimed at enhancing system consistency. Furthermore, particular emphasis should be placed on incorporating feedback from radiologists into the system’s design, ensuring that it aligns with their preferences and facilitates ease of use. Notably, our findings indicated that radiologists with fewer years of experience exhibited higher usability scores, highlighting their increased receptivity toward AI systems. This underscores the importance of considering the level of acceptability among radiologists when implementing AI technologies.^[21]

The learnability scores were rated as “OK,” indicating that radiologists perceived the system as reasonably trainable. However, radiologists also expressed a common opinion that they lacked sufficient training in the field of imaging AI. This underscores the importance of equipping radiologists with the requisite knowledge and awareness to facilitate their seamless adaptation to the system.^[8,22] Waymel et al. revealed that radiologists exhibited a deficiency in pre-existing knowledge and information pertaining to AI.^[7] Proficiency in advanced AI expertise can significantly augment the extent of AI’s clinical utilization, while a rudimentary understanding of AI can potentially hinder radiologists’ effective application of the technology.^[10] The absence of structured training frequently necessitates radiologists to engage in self-directed learning or resort to commercial videos and product manuals provided by software vendors to acquire the necessary knowledge and usage skills. This approach could present challenges as radiologists may not fully grasp the software’s functionality and applicability, which could potentially lead to an overestimation of the product’s performance.

Necessary decision support

Although the sample size is limited, our study provides in-depth insights, particularly regarding the impact of AI technology on the professional attitudes of radiologists. This aspect of the research is highly significant which, yet, remains underexplored. The AI system has exerted a beneficial influence on imaging departments, underlining the importance of its widespread adoption. Radiologists hold the belief that AI has played a pivotal role in enhancing their work efficiency and diagnostic accuracy. These favorable perspectives align closely with findings from prior research studies.^[23,24] AI serves as a valuable aid to junior radiologists in the diagnostic process, and its role in enhancing healthcare, particularly in underdeveloped regions, is of important significance.^[25] It is conceivable that junior radiologists may more readily embrace and adeptly utilize AI software compared to their senior counterparts. Nevertheless, the advantages of AI for senior radiologists appear to be constrained. The task of labeling false-positive results may result in increased physician workload and additional stress for patients.^[26]

The viewpoints of radiologists concerning the potential impact of AI on their future employment exhibit a complex feature. The evolution of AI confronts a multitude of challenges, including issues of uncertainty and non-interpretability, which preclude exclusive reliance on AI for medical diagnosis. Consequently, radiologists have assumed an indispensable role in light of these circumstances.^[27] Ideally, the integration of AI should commence within the medical curriculum from the outset of medical students’ education. This early exposure allows students to enhance their learning, acclimatize to future work environments, and cultivate a lasting sense of confidence in this evolving technology.^[28]

Limitations

Our study is subject to several limitations. First, the survey’s representativeness could be improved. We exclusively targeted radiologists from two tertiary hospitals, Beijing Anzhen Hospital, and Beijing Tsinghua Changgung Hospital, potentially limiting the generalizability of our findings to other health-care institutions. It is noteworthy that although these two hospitals had merged into the same company at the time of the survey, the AI systems they employed were developed by distinct companies, which introduces a degree of heterogeneity. In future research, we plan to increase the sample size to validate and expand on the current findings, particularly concerning the impact of AI on the future profession of radiologists. Second, the distribution of the survey by the heads of the imaging department to all radiologists may have introduced social desirability bias, as radiologists might have been inclined to provide positive evaluations due to this hierarchical relationship. Third, it is important to acknowledge that our study primarily elucidates overarching trends within the surveyed population. The considerable variance observed in the responses to each question underscores the persisting diversity in opinions among individual radiologists regarding this subject. In light of this diversity, future research may benefit from supplementing our findings with in-depth interviews to obtain a more comprehensive understanding of the intricate perspectives held by radiologists on this topic.