Translate this page into:
Evaluating the diagnostic performance of artificial intelligence-assisted decision-making software for pulmonary nodules in a resource-limited setting
*Corresponding author: Di Chen, Department of Rehabilitation Information Research, China Rehabilitation Science Institute, Beijing, China. cindino80@126.com
-
Received: ,
Accepted: ,
How to cite this article: Liao X, Tian Y, Cheng Y, Sun X, Li Y, Zhao Z, et al. Evaluating the diagnostic performance of artificial intelligence-assisted decision-making software for pulmonary nodule in a resource-limited setting: Insights from a secondary hospital in China. J Clin Imaging Sci. 2026;16:6. doi: 10.25259/JCIS_212_2025
Abstract
Objectives:
Emerging evidence suggested that artificial intelligence (AI) may offer particular benefits in resource-limited clinical settings with high patient loads and constrained radiology expertise. The present study aimed to evaluate the diagnostic performance of an AI-assisted decision-making software (DMS) for pulmonary nodules detected on computed tomography (CT) among physicians in a resource-limited clinical setting.
Material and Methods:
In this retrospective multi-reader, multi-case study, three pulmonologists and three radiologists from a secondary hospital independently assessed 200 enriched chest CT scans with and without AI-assisted DMS. The dataset was balanced with 100 benign and 100 malignant nodules to provide a consistent challenge for both physicians and the AI system. Diagnostic performance was measured by comparing the average area under the receiver operating characteristic curves (AUC) with and without AI support. Sensitivity and specificity were evaluated at the 5% and 65% malignancy thresholds, and inter-reader agreement on disease management plans was examined.
Results:
AI-assisted DMS significantly improved readers’ diagnostic performance, with the average AUC increasing from 0.78 to 0.89 (mean difference: 0.11, 95% confidence interval [CI]: 0.08, 0.14). Improvements were consistent across readers’ experience levels and specialties. Sensitivity at the 5% malignancy threshold reached 97.3% (95% CI: 95.1%, 99.6%) with AI assistance, while specificity improved by 18.5% (95% CI: 6.5%, 30.5%). At the 65% threshold, sensitivity and specificity increased by 21.2% and 7.8%, respectively. In addition, the overall inter-reader agreement enhanced from 0.19 to 0.40 (p < 0.01), although agreement on non-surgical diagnostic procedures remained relatively lower compared to other categories.
Conclusion:
AI-assisted DMS showed great potential in improving diagnostic performance for CT pulmonary nodule management in the resource-limited setting. Strengthening referral pathways for intermediate-risk cases might further support appropriate clinical decision-making and help align patient evaluation with available expertise. Continued prospective real-world studies with longitudinal follow-up and histopathological confirmation would contribute to expanding the evidence base and guiding its broader integration into routine clinical practice.
Keywords
Artificial intelligence
Clinical decision-making
Pulmonary nodules
Resource-limited setting
INTRODUCTION
Lung cancer is the leading cause of cancer-related deaths globally.[1] Early detection through low-dose computed tomography (CT) scans is essential for reducing mortality.[2,3] However, interpreting CT scans posed ongoing challenges due to variability in physicians’ training and experience, as well as the potential for human error, contributing to inconsistent diagnostic accuracy.[4,5]
The onset of the Coronavirus Disease 2019 pandemic increased public awareness of respiratory health worldwide, which substantially drove more cancer-related inquiries and greater uptake of cancer screening.[6-8] In China, in particular, low-dose CT screening expanded from targeting at-risk populations to becoming a standard part of nationwide health examinations, primarily due to the high accessibility and affordability of the scans.[9,10] Accordingly, more pulmonary nodules were detected, with a significant increase in detection rate from 46.9% before the pandemic to 57.6% afterward during routine check-ups.[11] Moreover, a growing number of lung cancer cases were identified in younger, non-smoking females, contrasting with the demographics of traditional “high-risk” populations.[10-12] The striking increase in detected nodules, particularly among non-high-risk individuals, introduced new challenges for lung cancer diagnosis, especially in resource-limited settings like secondary hospitals, which manage a large portion of the population-level screening. However, compared to tertiary and specialized hospitals, they often faced resource constraints, such as limited access to experienced physicians and advanced diagnostic technologies, negatively impacting the quality of care.[13-15] Accordingly, enhancing diagnostic capacity in these settings became crucial for improving clinical outcomes and expanding access to high-quality care.
Artificial intelligence (AI)-assisted decision-making software (DMS) provided great promises by enhancing diagnostic performance and clinical efficiency. AI algorithms were designed to support physicians by automatically identifying and risk-stratifying pulmonary nodules, leading to enhanced diagnostic performance, standardized management recommendations, and reduced reading times.[16-19] Effective implementation of AI-assisted DMS was fundamental for fully realizing its potential in clinical practice, yet integrating AI-assisted DMS into routine clinical workflows remained a major challenge.[20,21] In China, most AI-assisted DMS applications to date have been concentrated in well-resourced tertiary hospitals, where experienced radiologists and advanced infrastructure are already available.[22] Despite these favorable conditions, a stratified cluster survey of 160 tertiary hospitals across six provinces in China showed that only 23.8% had installed or tested AI software, underscoring the significant gaps in implementing AI-assisted diagnostic technologies within the healthcare system.[22]
This limited adoption highlights the gap between AI development and its broader integration, particularly in lower-tier hospitals where AI may provide even greater benefits.[20,23,24] Utilizing the updated Consolidated Framework of Implementation Research to qualitatively explore the barriers and facilitators, a prior study proposed that expanding AI implementation into primary and secondary care could maximize its clinical and system-level impact.[20] Building on this, the current study aimed to assess the performance of AI-assisted DMS for CT pulmonary nodules on physicians in a secondary hospital in China, providing additional insights to guide the scaling of AI integration across diverse healthcare environments.
MATERIALS AND METHODS
Design overview
A fully-crossed, retrospective, multi-reader, multi-case (MRMC) study was conducted, where readers evaluated CT scans with and without the aid of AI-assisted DMS. For patients with multiple pulmonary nodules, the dominant pulmonary nodule was assessed for malignancy, with a corresponding disease management plan developed, aligning with the existing guidelines to enhance study efficacy.[25,26]
Centralized ethical approval for the study was obtained from the Ethical Review Board of China Rehabilitation Research Center (approval number 2024-032-1), with informed consent waived for the retrospective evaluation of de-identified chest CT scans. The study was reported in accordance with the STARD 2015 (Standards for Reporting of Diagnostic Accuracy Studies) checklist to ensure transparency and completeness in diagnostic accuracy reporting.[27]
Study materials and settings
AI-assisted DMS
A commercially available AI-assisted screening system for pulmonary nodules (HY Medical Technology Co., Ltd, Beijing, China) was utilized. The system was approved by the National Medical Product Administration in 2022 and obtained CE certification in 2020. A proprietary 3D convolutional neural network (CNN) for object detection, utilizing random initialization, supervised training, and model ensemble techniques, was employed to accurately detect pulmonary nodules.
The underlying CNN algorithm was trained on a dataset of 4,638 chest CT scans from five hospitals across two provinces in China, including patients over 18 years old with a balanced sex distribution (female: 45.7%; male: 46.2%; missing: 8.1%). The CT imaging equipment used in the dataset was primarily from Siemens (44.1%), GE (37.8%), and Philips (11.7%), reflecting typical clinical usage. All scans had a slice thickness below 2.5 mm, with most between 1 mm and 2 mm . The AI-assisted DMS automatically located and segmented all potential lesions in each CT scan and provided a corresponding malignancy score ranging from 0% to 100%. The core functions of the AI-assisted DMS are shown in Table 1.
| Function | Description |
|---|---|
| Automatic detection | Analyzes thin-slice lung CT images to detect and annotate pulmonary nodules, providing a one-click guidance function to support clinical decision-making. |
| Automatic measurement | Accurately segments nodule contours, automatically measuring lesion diameter, volume, density, surface area, and histogram information |
| Automatic characterization | Automatically classifies the nature of the nodule, including solid, part-solid, ground-glass, and calcified nodules |
| Automatic diagnosis | Predicts the risk level of the nodule (high, medium, low) based on international guidelines and provides the probability of malignancy |
| Automatic navigation | Supports MPR (Multi-Planar Reconstruction) display for simultaneous sagittal, coronal, and axial views |
| One-Click report generation | Generates a structured textual description and imaging conclusions with a single click |
| Professional follow-Up suggestion | Provides follow-up suggestions according to international guidelines. |
Settings
A total of 200 de-identified data were collected from two research sites: Site A, a secondary hospital specializing in rehabilitation, and Site B, a general tertiary hospital [Table 2].
| Site A | Site B | |
|---|---|---|
| Name | Fengtai Rehabilitation Hospital of Beijing (Tieying Hospital) | China Rehabilitation Research Center (Beijing Boai Hospital) |
| Location | Beijing | Beijing |
| Funding source | Public | Public |
| Tier | Secondary | Tertiary |
| Hospital type | Specialized | General |
| Specialties | Neurological, geriatric, bone and joint rehabilitation services, etc. | Specialized respiratory, sports injury, hearing and speech rehabilitation services, etc. |
| Bed capacity | 211 | 1,100 |
| Staff number | 434 | 1,700 |
| Annual outpatient volume | 300,000 | 470,000 |
Reference standard
Histopathological confirmation remains the gold standard for determining the benign or malignant nature of pulmonary nodules. However, in the absence of consistently accessible histopathological confirmation, a pragmatic reference standard was established using an expert consensus approach, reflecting routine clinical practice in resource-limited settings in China. In real-world care pathways, patients with suspected malignant pulmonary nodules detected at secondary hospitals or non-specialized tertiary hospitals often undergo further diagnostic evaluation or treatment at specialized cancer hospitals. However, due to the lack of interoperability between medical information systems across institutions, histopathological results are frequently unavailable for retrospective analyses at the referring hospital. More importantly, restricting the study population to cases with available pathological confirmation would have introduced substantial selection bias, disproportionately enriching the sample with higher-risk nodules.
Accordingly, two experienced consultant pulmonologists from Site B (with 24 and 10 years of experience, respectively) independently evaluated the dominant pulmonary nodule in each CT scan, using all relevant clinical information but without access to AI-generated results. Disagreements were resolved by a thorough discussion with a third expert radiologist (with 14 years of experience). Beyond malignancy assessment, the consultants were asked to rate the difficulty of each case on a 5-point Likert scale, where 1 indicated “very easy” and 5 indicated “very difficult.” The mean difficulty score for each case was calculated, and cases were further categorized into three groups, including easy (1 ≤ mean score < 3), moderate (mean score = 3), and challenging (3 < mean score ≤ 5).
Data selection
No imaging data from the training dataset used to develop the AI-assisted DMS was included in this study. A total of 799 de-identified CT scans were retrospectively collected from two research sites [Figure 1]. Obtained from inpatient, outpatient, and emergency settings, the CT scans were preliminarily pre-screened by two radiologists who were not involved in the reader study, using the following pre-specified eligibility criteria. The inclusion criteria were as follows:
- Flowchart of the selection process of computed tomography scan images and an overview of the reader study.
Digital imaging and communications in medicine (DICOM) image formats;
Chest CT scans only;
Patients aged ≥ 18 years;
Largest axial diameter of the dominant nodule between 4 and 25 mm;
Slice spacing ≤ 2.5 mm.
The exclusion criteria were as follows:
Patients with prior lung surgery;
Presence of image artifacts;
Calcified nodules.
After further screening, 66 CT scans were excluded [Figure 1]. From the remaining eligible scans, 200 CT scans were selected consecutively, until the required number of cases and desired enrichment in case characteristics were reached. To provide a consistent challenge for both physicians and the AI system, the final dataset was enriched to achieve a 50% prevalence of malignancy and an approximately balanced distribution of case difficulty, with a 1:1 ratio between easy cases and moderate or challenging cases.
Reader study
An overview of the reader study design has been shown in Figure 1. Six physicians from the secondary hospital (Site A) participated, including three accredited radiologists and three accredited pulmonologists. The readers were stratified by level of clinical experience into two groups: Three senior readers and three non-senior readers [Table 3].
| Reader | Level of clinical experience | Specialty | Sex |
|---|---|---|---|
| R1 | Senior | Pulmonology | Female |
| R2 | Senior | Pulmonology | Male |
| R3 | Non-senior | Pulmonology | Male |
| R4 | Non-senior | Radiology | Female |
| R5 | Non-senior | Radiology | Male |
| R6 | Senior | Radiology | Female |
All readers conducted clinical diagnoses using a web-based DICOM viewer on standard hospital workstations under typical reading conditions. Each CT scan was loaded into the viewer with the nodule of interest highlighted. The readers, blinded to the available clinical information, reference standards, and the prevalence of malignancy, reviewed 200 CT scans in random order, both with and without AI-assisted DMS. The DICOM viewer performed the randomization. Each CT scan was interpreted by sequentially scrolling through the images to make a clinical diagnosis, with no time limit imposed.
For each CT scan, readers assessed the highlighted pulmonary nodule by assigning a malignancy risk score from 1 to 100. Following the guidelines of the American College of Chest Physicians, nodules with risk under 5% were considered low-risk, 5–65% intermediate-risk, and above 65% high-risk.[26] In addition, for each case, readers assigned a disease management plan from one of the three categories: CT surveillance (short-term or long-term follow-up), non-surgical diagnostic procedures (Positron emission tomography [PET]/CT or non-surgical biopsy), or surgical resection.
Before the official reading sessions, all readers received comprehensive training and practiced on 10 cases that were not included in the final dataset. In the first AI-unaided session, readers assessed 200 CT scans without the use of AI-assisted DMS. After a 4-week washout period, the AI-aided session was conducted, during which the AI-estimated malignancy risk scores were displayed on the viewer to support re-evaluation of each case.
Outcomes
The primary outcome was the difference in the readers’ average area under the receiver operating characteristics curves (AUC) between the AI-aided and unaided sessions. Secondary outcomes included differences in sensitivity and specificity at the malignancy risk thresholds of 5% and 65%. In addition, the inter-reader agreement on disease management plans was evaluated.
Statistical analysis
The null hypothesis was that there was no difference between the average AUC for the AI-aided and unaided reading modalities. Based on existing literature and pilot study results using a similar AI-assisted DMS for CT pulmonary nodule diagnosis, a sample of six readers and a minimum of 173 chest CT scans was required to detect a minimal AUC difference of 0.07, with a statistical power of 0.8 and a type I error rate of 0.05.[18,28,29] The sample size was further expanded to 200 CT scans.
The Obuchowski-Rockette (OR) method for MRMC analysis was used to calculate the mean difference in readers’ AUC between the AI-aided and unaided sessions, along with the corresponding 95% confidence interval (CI). A two-way mixed-effects analysis of variance model was employed, treating readers and cases as random effects. Covariances were estimated using the DeLong method. Subgroup analyses were performed to examine differences in average AUC, stratified by reader specialty, level of clinical experience, nodule size, and nodule density. Nodule size was dichotomized into nodules with the largest axial diameter of 4–10 mm and those of 10–25 mm. Nodule density was recategorized into either solid or sub-solid, with sub-solid nodules including part-solid and pure ground-glass types. Differences in sensitivity and specificity between the two modalities were assessed using the OR model with the jackknife method. Secondary outcomes were tested for significance only if the null hypothesis for the primary outcome was rejected. Fleiss’ kappa was utilized to evaluate multi-reader agreement on disease management plans. The interpretation of the Kappa statistics has been shown in Table 4.[30]
| Kappa value | Strength of agreement |
|---|---|
| 0–0.2 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.0 | Excellent |
A two-sided p < 0.05 was considered statistically significant. All statistical analyses were conducted using the MRMCaov package for R Studio (version 2024.09.0 + 375).
RESULTS
Case characteristics
A total of 200 chest CT images were collected from 200 patients, including 100 with benign and 100 with malignant pulmonary nodules. The final data set consisted of 89 easy cases and 111 moderate or challenging cases. The median age of participants was 66 years (Interquartile Range [IQR]: 53 – 76.2 years), and the majority were female (53.5%) [Table 5]. Of the pulmonary nodules, 120 (60%) measured between 4 and 10 mm, and 54.5% were solid. As compared to benign pulmonary nodules, malignant nodules were larger in size and more frequently sub-solid (including part-solid nodules and pure ground-glass nodules).
| Characteristics | Total (n=200) | Benign (n=100) | Malignant (n=100) |
|---|---|---|---|
| Age (year)a | 66 (53–76.2) | 66 (51–76.2) | 66 (57–76.2) |
| Sex | |||
| Male | 93 (46.5%) | 46 (46%) | 47 (47%) |
| Female | 107 (53.5%) | 54 (54%) | 53 (53%) |
| Nodule diameter (mm) | |||
| 4–<10 | 120 (60%) | 81 (81%) | 39 (39%) |
| 10–25 | 80 (40%) | 19 (19%) | 61 (61%) |
| Nodule density | |||
| Solid | 109 (54.5%) | 67 (67%) | 42 (42%) |
| Part-solid | 30 (15%) | 8 (8%) | 22 (22%) |
| Ground-glass | 61 (30.5%) | 25 (25%) | 36 (36%) |
a: Median (IQR). CT: Computed tomography, IQR: Interquartile range
Reader performance
The average AUC for discriminating pulmonary nodule malignancy among the six readers significantly increased from 0.78 (95% CI: 0.73, 0.83) without AI assistance to 0.89 (95% CI: 0.85, 0.93) with AI assistance, reflecting a mean difference of 0.11 (95% CI: 0.08, 0.14) [Figure 2].
- Average area under the receiver operating characteristic curve across all readers in discriminating pulmonary nodule malignancy without and with the aid of artificial intelligence-assisted decision-making software.
Furthermore, AI-assisted DMS consistently and significantly enhanced reader-level diagnostic performance across all specialties and experience levels [Table 6]. The smallest increase in AUC was observed in a senior radiologist, with an improvement of 0.08 (95% CI: 0.03, 0.13). Conversely, the largest improvement occurred in a non-senior pulmonologist, whose AUC increased from 0.74 (95% CI: 0.67, 0.81) to 0.88 (95% CI: 0.83, 0.92).
| Reader | AUC without AI (95% CI) | AUC with AI (95% CI) | Delta AUC (95% CI) |
|---|---|---|---|
| 1 | 0.78 (0.71, 0.84) | 0.87 (0.82, 0.92) | 0.09 (0.02, 0.16) |
| 2 | 0.76 (0.70, 0.83) | 0.88 (0.82, 0.93) | 0.11 (0.04, 0.18) |
| 3 | 0.74 (0.67, 0.81) | 0.88 (0.83, 0.92) | 0.13 (0.07, 0.19) |
| 4 | 0.77 (0.70, 0.84) | 0.89 (0.85, 0.94) | 0.12 (0.06, 0.19) |
| 5 | 0.80 (0.74, 0.86) | 0.92 (0.88, 0.96) | 0.12 (0.06, 0.17) |
| 6 | 0.83 (0.77, 0.89) | 0.91 (0.87, 0.95) | 0.08 (0.03, 0.13) |
AUC: Area under the receiver operating characteristics curves, AI: Artificial intelligence, CI: Confidence interval
Subgroup analyses further confirmed that both pulmonologists and radiologists gained comparable improvements in discriminating pulmonary nodule malignancy with AI-assisted DMS [Table 7]. Non-senior physicians demonstrated a modestly greater improvement in AUC compared to senior physicians, although the difference was not statistically significant. For pulmonary nodules measuring between 4 mm and 10 mm, AI assistance significantly increased the AUC to 0.90 (95% CI: 0.85, 0.95). Notably, AI-assisted DMS provided a significantly greater improvement in AUC for sub-solid nodules than for solid nodules (p = 0.021). Representative case examples of AI predictions paired with corresponding reference standard diagnoses have been shown in Figure 3.
| Characteristics | AUC without AI (95% CI) | AUC with AI (95% CI) | Delta AUC (95% CI) |
|---|---|---|---|
| Specialty | |||
| Pulmonology | 0.76 (0.71, 0.81) | 0.87 (0.83, 0.91) | 0.11 (0.06, 0.16) |
| Radiology | 0.80 (0.74, 0.86) | 0.91 (0.87, 0.94) | 0.11 (0.07, 0.15) |
| Level of experience | |||
| Non-senior | 0.77 (0.72, 0.83) | 0.90 (0.85, 0.94) | 0.12 (0.10, 0.15) |
| Senior | 0.79 (0.72, 0.86) | 0.89 (0.84, 0.93) | 0.10 (0.06, 0.13) |
| Nodule size (mm) | |||
| 4–<10 | 0.78 (0.72, 0.84) | 0.90 (0.85, 0.95) | 0.12 (0.06, 0.18) |
| 10–25 | 0.62 (0.50, 0.74) | 0.74 (0.62, 0.86) | 0.12 (0.01, 0.23) |
| Nodule density* | |||
| Solid | 0.86 (0.82, 0.91) | 0.92 (0.88, 0.96) | 0.06 (0.02, 0.09) |
| Sub-Solida | 0.69 (0.62, 0.78) | 0.84 (0.76, 0.92) | 0.14 (0.08, 0.20) |
a: Sub-solid pulmonary nodules included part-solid and pure ground-glass nodules. *p<0.05. AUC: Area under the receiver operating characteristics curves, AI: Artificial intelligence, CI: Confidence interval
- Representative computed tomography (CT) lung nodule cases showing artificial intelligence (AI)-assisted predictions paired with final diagnoses. CT image of a 55-year-old male with the nodule of interest indicated by a red circle. The yellow square represents the AI-predicted malignancy probability (99%) and the corresponding high-risk category. The final diagnosis, established by the reference standard, confirmed malignancy. (b) CT image of a 62-year-old female with the nodule of interest indicated by a red circle. The yellow square represents the AI-predicted malignancy probability (0%) and the corresponding low-risk category. The final diagnosis, established by the reference standard, confirmed benignity.
Secondary outcomes
As shown in Table 8, the average sensitivity at the 5% malignancy threshold increased significantly with AI assistance, reaching 97.3% (584/600, 95% CI: 95.1%, 99.6%), which was 6.2% higher than without AI assistance (91.2%, 95% CI: 86.6%, 95.6%). Average specificity also revealed significant enhancement, increasing from 40.2% (241/600) to 58.7% (352/600), with a difference of 18.5% (95% CI: 6.5%, 30.5%).
| Performance indicator | 5% Malignancy probability threshold | 65% Malignancy probability threshold | ||||
|---|---|---|---|---|---|---|
| Without AI | With AI | Delta | Without AI | With AI | Delta | |
| True positives (n) | 547 | 584 | 288 | 415 | ||
| False negatives (n) | 53 | 16 | 312 | 185 | ||
| True negatives (n) | 241 | 352 | 491 | 538 | ||
| False positives (n) | 359 | 248 | 109 | 62 | ||
| Sensitivity (%) | 91.2 (86.8, 95.6) | 97.3 (95.1, 99.6) | 6.2 (1.6, 10.7) | 48.0 (23.3, 72.7) | 69.2 (52.9, 85.4) | 21.2 (0.2, 42.1) |
| Specificity (%) | 40.2 (24.8, 55.6) | 58.7 (47.8, 69.5) | 18.5 (6.5, 30.5) | 81.8 (70.6, 93.1) | 89.7 (83.1, 96.2) | 7.8 (1.6, 14.1) |
| False-negative rate (%) | 8.8 (4.4, 13.2) | 2.7 (0.4, 4.9) | −6.2 (−10.7, −1.6) | 52.0 (27.3, 76.7) | 30.8 (14.6, 47.1) | −21.2 (−42.1, −0.2) |
| False-positive rate (%) | 59.8 (44.4, 52.2) | 41.3 (30.5, 75.2) | −18.5 (−30.5, −6.5) | 18.2 (6.9, 29.4) | 10.3 (3.8, 16.9) | −7.8 (−14.1, −1.5) |
AI: Artificial intelligence, DMS: Decision-making software, values in the bracket represent : 95% confidence interval
Similarly, at the 65% malignancy threshold, AI assistance was associated with incremental improvements on both sensitivity and specificity, with increases of 21.2% (95% CI: 0.2%, 42.1%) and 7.8% (95% CI: 1.6%, 14.1%), respectively.
Reader agreement
The use of AI-assisted DMS significantly enhanced overall inter-reader agreement on disease management decisions, increasing from 0.19 (95% CI: 0.12, 0.25) without AI assistance to 0.40 (95% CI: 0.34, 0.46) with AI assistance (p < 0.01) [Table 9]. Across all disease management sub-categories, AI assistance was associated with consistent and significant improvements in inter-reader agreement. Agreement for CT surveillance increased from 0.28 to 0.55, and for surgical resection from 0.15 to 0.40 (p < 0.01 for both). Although AI assistance significantly increased agreement for non-surgical diagnostic procedures from 0.07 to 0.19 (p < 0.01), the level of agreement remained lower compared to other categories.
| Disease management category | Without AI | With AI | p-value |
|---|---|---|---|
| CT surveillancea | 0.28 (95% CI: 0.23, 0.34) | 0.55 (95% CI: 0.50, 0.60) | <0.01 |
| Non-surgical diagnostic proceduresb | 0.07 (95% CI: 0.01, 0.13) | 0.19 (95% CI: 0.12, 0.25) | <0.01 |
| Surgical resection | 0.15 (95% CI: 0.09, 0.21) | 0.40 (95% CI: 0.35, 0.46) | <0.01 |
| Overall | 0.19 (95% CI: 0.12, 0.25) | 0.40 (95% CI: 0.34, 0.46) | <0.01 |
a: CT surveillance includes short-term or long-term follow-up, b: Non-surgical diagnostic procedures include PET/CT or non-surgical biopsy. AI: Artificial intelligence, CI: Confidence interval, CT: Computed tomography, PET: Positron emission tomography. A p-value < 0.05 was considered statistically significant, Value in the bracket: confidence interval
DISCUSSION
This MRMC study is the first to evaluate the diagnostic performance of an AI-assisted DMS for CT pulmonary nodule assessment in a resource-limited setting in China. Results demonstrated a significant improvement in average AUC discriminating malignant pulmonary nodules, with an increase of 0.11 (95% CI: 0.08, 0.14). This improvement was consistent with existing literature, where an increase in AUC ranged from 0.02 to 0.13.[18,31-33] Notably, subgroup analyses suggested that both radiologists and pulmonologists, irrespective of their level of experience, benefited from AI assistance. Non-senior physicians had a modestly greater AUC improvement (0.12, 95% CI: 0.10, 0.15) compared to senior physicians (0.10, 95% CI: 0.06, 0.13), with individual reader-level improvements ranging from 0.08 to 0.13, demonstrating the added value of AI-assisted DMS across varying levels of expertise in secondary care. Both sensitivity and specificity at clinically relevant malignancy thresholds increased significantly with AI assistance. At the 5% malignancy threshold, AI-assisted DMS supported physicians at secondary care by reducing the number of missed malignant cases and decreasing the false-negative rate by 6.2% (95% CI: −10.7%, −1.6%). Likewise, at the 65% malignancy threshold, specificity increased significantly to 89.7% (95% CI: 83.1%, 96.2%), highlighting the potential of AI-assisted DMS in helping physicians rule out malignancy in high-risk patients.
AI-assisted DMS has been well-established as an effective tool for training junior physicians in diagnosis and clinical decision-making across various therapeutic areas, including pulmonology,[19,34,35] thyroid diseases,[36,37] dentistry,[38] ophthalmology,[39,40] gastroenterology,[41,42] orthopedics,[43-45] and oncology.[46] In contrast, the benefit of AI-assisted DMS for senior physicians remained unclear. Some studies reported no significant gains for senior physicians,[36,45,47,48] while others found modest improvements in pulmonary nodule detection and malignancy classification.[49,50] In addition, the clinical value of AI-assisted DMS tended to be limited to cases involving poorly distinguishable characteristics or to enhancing workflow efficiency.[19,50-52] These mixed results stressed the challenges of effectively integrating AI-assisted DMS into clinical practice.
It should be noted that studies reporting comparable diagnostic performance between senior physicians with and without AI-assisted DMS often involved those in general tertiary hospitals or specialized cancer centers, where diagnostic skills were highly developed.[36,45,48] Consequently, the practical benefits of AI-assisted DMS for physicians with high baseline diagnostic performance became uncertain.[49] In these settings, AI-assisted DMS aligned more with training needs than supporting senior physicians who primarily deal with more complex and specialized conditions. Shifting implementation efforts toward primary or secondary healthcare settings, where AI-assisted DMS could proficiently address key diagnostic challenges, may fully leverage its potential.[20,23,24,53-55] However, evidence on the clinical performance in primary and secondary settings remained limited, as most existing studies were exploratory. Employing post-visit questionnaires, Wiedermann et al. (2023) revealed that only 27% of participating general practitioners were highly satisfied with an AI-driven symptom checker in Italian primary care, with concerns about low accuracy and misdiagnosis.[53] The present study was the first to evaluate the diagnostic performance of AI-assisted DMS for physicians of varying specialties and clinical experience in a secondary hospital setting, providing preliminary evidence supporting AI-assisted DMS implementation at primary and secondary care, particularly in China.
Furthermore, AI-assisted DMS improved inter-reader agreement on disease management plans, enhancing consistency among physicians. Overall inter-reader agreement increased significantly from 0.19 to 0.40 (P < 0.01) with AI support, reflecting a shift from slight to fair agreement across all readers. Specifically, AI-assisted DMS enhanced inter-reader agreement from fair to moderate for CT surveillance and from slight to fair for surgical resection, leading to more standardized decision-making among physicians. This result aligned with previous studies showing that AI-assisted DMS reduced discrepancies in managing low- and high-risk pulmonary nodules.[18,56] However, AI-assisted inter-reader agreement for non-surgical diagnostic procedures remained low at 0.19 (95% CI: 0.12, 0.25), indicating only slight agreement. Typically, non-surgical diagnostic procedures, including PET scan and non-surgical biopsies, are recommended for patients with an intermediate malignancy risk.[26] The low agreement suggested marked heterogeneity in managing intermediate-risk nodules among physicians at secondary hospitals, likely due to the inherent complexity of these cases, which often required individualized decision-making. Clinically, the intermediate-risk category represented the most challenging, with recommendations ranging from surveillance to biopsy or resection depending on clinical guidelines and patient preferences.[57,58] In secondary hospitals, inconsistencies in managing intermediate-risk nodules were particularly obvious, which might be associated with the high prevalence of low-risk nodules in the general population served by the hospital.[59] However, effectively and appropriately addressing the management of intermediate-risk nodules remained critical, as these cases presented unique challenges. A community-based study indicated that a quarter of intermediate pulmonary nodules referred to a pulmonologist were malignant.[60] Given this diagnostic uncertainty, establishing structured referral pathways for intermediate-risk patients became crucial, allowing tertiary hospitals to prioritize patients with complex diagnostic needs and ensuring that challenging cases receive specialized care.[20] In China, however, inter-hospital referral systems were still under development, posing considerable challenges to timely and coordinated care.[61-63] Fully leveraging the potential of AI-assisted DMS relied on a well-designed referral mechanism, directing complex cases from secondary hospitals to tertiary hospitals with advanced diagnostic resources and specialized expertise. Future studies should explore effective referral strategies, such as collaborative networks and integrated information platforms, to strengthen the integration of AI-assisted DMS into the healthcare system.
CONCLUSION
The findings of this study highlighted the substantial potential of AI-assisted DMS to enhance diagnostic performance and promote standardized disease management in resource-limited settings in China. The observed patterns in the management of intermediate-risk nodules underscored the importance of establishing structured referral pathways to tertiary hospitals, where complex cases can benefit from advanced diagnostic resources and specialized expertise. To maximize the value of AI-assisted DMS across diverse healthcare environments, continued efforts to optimize referral mechanisms will help support consistent, high-quality care for patients requiring additional evaluation. Further prospective studies involving real-world patient cohorts, with systematic longitudinal follow-up and access to histopathological confirmation where clinically indicated, are needed to fully evaluate its clinical applicability and integration into routine practice.
Acknowledgments:
The authors acknowledge HY Medical Technology Co., Ltd., for providing the AI-assisted DMS and DICOM web viewer, Yingzhe Fu, Danrui Zong, Yuanyuan Lun, and Shentang Wang for participation in imaging data interpretation, Ling Chen, Qimin Wang, and Hongxia Zhang for contributions to the reference standard, and Meixia Shang for valuable suggestions on the manuscript.
Ethical approval:
The research/study approved by the Institutional Review Board at Ethical Review Board of China Rehabilitation Research Center, number 2024-032-1, dated June 25, 2024.
Declaration of patient consent:
Patient consent was not required as there are no patients in this study.
Conflicts of interest:
There are no conflicts of interest.
Use of artificial intelligence (AI)-assisted technology for manuscript preparation:
The authors confirm that they have used artificial intelligence (AI)-assisted technology for assisting in the writing or editing of the manuscript or image creations. AI-assisted tools were used throughout the entire manuscript solely for language polishing and grammatical refinement. The AI was not involved in study design, data collection, data analysis, interpretation of results, or figure generation.
Financial support and sponsorship: Fundamental Research Funds for Central Public Welfare Research Institutes, conducted by China Rehabilitation Science Institute (CRSI2024CZ-1).
References
- Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2024;74:229-63.
- [CrossRef] [PubMed] [Google Scholar]
- Chinese Medical Association guideline for clinical diagnosis and treatment of lung cancer (2024 edition) J Chin Med Assoc. 2024;104:3175-213.
- [Google Scholar]
- Low-dose computed tomography screening reduces lung cancer mortality. Adv Med Sci. 2018;63:230-6.
- [CrossRef] [PubMed] [Google Scholar]
- Software using artificial intelligence for nodule and cancer detection in CT lung cancer screening: Systematic review of test accuracy studies. Thorax. 2024;79:1040-9.
- [CrossRef] [PubMed] [Google Scholar]
- Deep learning-based artificial intelligence improves accuracy of error-prone lung nodules. Int J Med Sci. 2022;19:490-8.
- [CrossRef] [PubMed] [Google Scholar]
- Cancer screening information-seeking before and during the COVID-19 pandemic. Prevent Oncol Epidemiol. 2024;2:2369168.
- [CrossRef] [Google Scholar]
- The influence of COVID-19 pandemic on the health seeking behaviors of adults living with chronic conditions: A view through the health belief model. J Prim Care Community Health. 2023;14:21501319231159459.
- [CrossRef] [PubMed] [Google Scholar]
- Changes in uptake of stool-based colorectal cancer screening during the Covid-19 pandemic. Cancer Causes Control. 2023;34:887-95.
- [CrossRef] [PubMed] [Google Scholar]
- Results of low-dose computed tomography as a regular health examination among Chinese hospital employees. J Thorac Cardiovasc Surg. 2020;160:824-31.e4.
- [CrossRef] [PubMed] [Google Scholar]
- Lung cancer screening: Who pays? Who receives?-The Chinese perspective. Transl Lung Cancer Res. 2021;10:2389-94.
- [CrossRef] [PubMed] [Google Scholar]
- "Low-age, low-frequency" lung cancer screening strategies maybe adaptable to the situation in China. BMC Surg. 2023;23:367.
- [CrossRef] [PubMed] [Google Scholar]
- Should nonsmokers be excluded from early lung cancer screening with low-dose spiral computed tomography? Community-based practice in Shanghai. Transl Oncol. 2017;10:485-90.
- [CrossRef] [PubMed] [Google Scholar]
- Quality of care in Chinese hospitals: Processes and outcomes after ST-segment elevation myocardial infarction. J Am Heart Assoc. 2017;6:e005040.
- [CrossRef] [PubMed] [Google Scholar]
- Healthcare quality and mortality among patients hospitalized for heart failure by hospital level in Beijing, China. ESC Heart Fail. 2021;8:1186-94.
- [CrossRef] [PubMed] [Google Scholar]
- Can a multitiered copayment system affect people's healthcare-seeking behavior? A case study of Wenzhou, China. BMC Health Serv Res. 2022;22:630.
- [CrossRef] [PubMed] [Google Scholar]
- Validation of a deep learning computer aided system for CT based lung nodule detection, classification, and growth rate estimation in a routine clinical population. PLoS One. 2022;17:e0266799.
- [CrossRef] [PubMed] [Google Scholar]
- Evaluating the performance of a deep learning-based computer-aided diagnosis (DL-CAD) system for detecting and characterizing lung nodules: Comparison with the performance of double reading by radiologists. Thorac Cancer. 2019;10:183-92.
- [CrossRef] [PubMed] [Google Scholar]
- Artificial intelligence tool for assessment of indeterminate pulmonary nodules detected with CT. Radiology. 2022;304:683-91.
- [CrossRef] [PubMed] [Google Scholar]
- An artificial intelligence-based chest X-ray model on human nodule detection accuracy from a multicenter study. JAMA Netw Open. 2021;4:e2141096.
- [CrossRef] [PubMed] [Google Scholar]
- Barriers and facilitators to implementing imaging-based diagnostic artificial intelligence-assisted decision-making software in hospitals in China: A qualitative study using the updated Consolidated Framework for Implementation Research. BMJ Open. 2024;14:e084398.
- [CrossRef] [PubMed] [Google Scholar]
- Artificial intelligence implementation in healthcare: A theory-based scoping review of barriers and facilitators. Int J Environ Res Public Health. 2022;19:16359.
- [CrossRef] [PubMed] [Google Scholar]
- Status of AI-enabled clinical decision support systems implementations in China. Methods Inf Med. 2021;60:123-32.
- [CrossRef] [PubMed] [Google Scholar]
- The artificial intelligence revolution in primary care: Challenges, dilemmas and opportunities. Aten Primaria. 2024;56:102820.
- [CrossRef] [PubMed] [Google Scholar]
- Using artificial intelligence to improve primary care for patients and clinicians. JAMA Intern Med. 2024;184:343-4.
- [CrossRef] [PubMed] [Google Scholar]
- Guidelines for management of incidental pulmonary nodules detected on CT images: From the fleischner society 2017. Radiology. 2017;284:228-43.
- [CrossRef] [PubMed] [Google Scholar]
- Evaluation of individuals with pulmonary nodules: When is it lung cancer? Diagnosis and management of lung cancer 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest. 2013;143:S93-120.
- [CrossRef] [PubMed] [Google Scholar]
- STARD 2015 guidelines for reporting diagnostic accuracy studies: Explanation and elaboration. BMJ Open. 2016;6:e012799.
- [CrossRef] [PubMed] [Google Scholar]
- Estimation and application of sample size in clinical trials of AI assisted medical products by multi-reader multi-case design. Chin J Health Stat. 2022;39:14-8.
- [Google Scholar]
- Multi-reader multi-case studies using the area under the receiver operator characteristic curve as a measure of diagnostic accuracy: Systematic review with a focus on quality of data reporting. PLoS One. 2014;9:e116018.
- [CrossRef] [PubMed] [Google Scholar]
- The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-74.
- [CrossRef] [Google Scholar]
- Deep learning for the classification of small (=2 cm) pulmonary nodules on CT imaging: A preliminary study. Acad Radiol. 2020;27:e55-63.
- [CrossRef] [PubMed] [Google Scholar]
- Effect of human-AI interaction on detection of malignant lung nodules on chest radiographs. Radiology. 2023;307:e222976.
- [CrossRef] [PubMed] [Google Scholar]
- The effects of artificial intelligence assistance on the radiologists' assessment of lung nodules on CT scans: A systematic review. J Clin Med. 2023;12:3536.
- [CrossRef] [PubMed] [Google Scholar]
- Artificial intelligence-based graded training of pulmonary nodules for junior radiology residents and medical imaging students. BMC Med Educ. 2024;24:740.
- [CrossRef] [PubMed] [Google Scholar]
- Artificial intelligence for assistance of radiology residents in chest CT evaluation for COVID-19 pneumonia: A comparative diagnostic accuracy study. Acta Radiol. 2023;64:2104-10.
- [CrossRef] [PubMed] [Google Scholar]
- The clinical value of artificial intelligence in assisting junior radiologists in thyroid ultrasound: A multicenter prospective study from real clinical practice. BMC Med. 2024;22:293.
- [CrossRef] [PubMed] [Google Scholar]
- A comparison of the performances of artificial intelligence system and radiologists in the ultrasound diagnosis of thyroid nodules. Curr Med Imaging. 2022;18:1369-77.
- [CrossRef] [PubMed] [Google Scholar]
- Comparison of artificial intelligence vs. junior dentists' diagnostic performance based on caries and periapical infection detection on panoramic images. Quant Imaging Med Surg. 2023;13:7494-503.
- [CrossRef] [PubMed] [Google Scholar]
- The performance of a deep learning system in assisting junior ophthalmologists in diagnosing 13 major fundus diseases: A prospective multi-center clinical trial. NPJ Digit Med. 2024;7:8.
- [CrossRef] [PubMed] [Google Scholar]
- The effectiveness of artificial intelligence-based automated grading and training system in education of manual detection of diabetic retinopathy. Front Public Health. 2022;10:1025271.
- [CrossRef] [PubMed] [Google Scholar]
- Feedback from artificial intelligence improved the learning of junior endoscopists on histology prediction of gastric lesions. Endosc Int Open. 2020;8:E139-46.
- [CrossRef] [PubMed] [Google Scholar]
- Artificial intelligence-based diagnosis of abnormalities in small-bowel capsule endoscopy. Endoscopy. 2023;55:44-51.
- [CrossRef] [PubMed] [Google Scholar]
- Clinical evaluation of AI software for rib fracture detection and its impact on junior radiologist performance. Acta Radiol. 2022;63:1535-45.
- [CrossRef] [PubMed] [Google Scholar]
- Comparison of diagnostic performance of a deep learning algorithm, emergency physicians, junior radiologists and senior radiologists in the detection of appendicular fractures in children. Pediatr Radiol. 2023;53:1675-84.
- [CrossRef] [PubMed] [Google Scholar]
- Assessment of an artificial intelligence aid for the detection of appendicular skeletal fractures in children and young adults by senior and junior radiologists: Reply to Sammer et al. Pediatr Radiol. 2023;53:341-2.
- [CrossRef] [PubMed] [Google Scholar]
- Artificial intelligence applications in pediatric oncology diagnosis. Explor Target Antitumor Ther. 2023;4:157-69.
- [CrossRef] [PubMed] [Google Scholar]
- The value of the Demetics ultrasound-assisted diagnosis system in the differential diagnosis of benign from malignant thyroid nodules and analysis of the influencing factors. Eur Radiol. 2021;31:7936-44.
- [CrossRef] [PubMed] [Google Scholar]
- Prospective study of AI-assisted prediction of breast malignancies in physical health examinations: Role of off-the-shelf AI software and comparison to radiologist performance. Front Oncol. 2024;14:1374278.
- [CrossRef] [PubMed] [Google Scholar]
- Generalizability and diagnostic performance of AI models for thyroid US. Radiology. 2023;307:e221157.
- [CrossRef] [PubMed] [Google Scholar]
- Using AI to improve radiologist performance in detection of abnormalities on chest radiographs. Radiology. 2023;309:e230860.
- [CrossRef] [PubMed] [Google Scholar]
- An artificial intelligence ultrasound system's ability to distinguish benign from malignant follicular-patterned lesions. Front Endocrinol (Lausanne). 2022;13:981403.
- [CrossRef] [PubMed] [Google Scholar]
- Integration of artificial intelligence decision aids to reduce workload and enhance efficiency in thyroid nodule management. JAMA Netw Open. 2023;6:e2313674.
- [CrossRef] [PubMed] [Google Scholar]
- Redesigning primary care: the emergence of artificial-intelligence-driven symptom diagnostic tools. J Pers Med. 2023;13:1379.
- [CrossRef] [PubMed] [Google Scholar]
- A clinician's guide to artificial intelligence (AI): Why and how primary care should lead the health care AI revolution. J Am Board Fam Med. 2022;35:175-84.
- [CrossRef] [PubMed] [Google Scholar]
- Artificial intelligence for early diagnosis of lung cancer through incidental nodule detection in low-and middle-income countries-acceleration during the COVID-19 pandemic but here to stay. Am J Cancer Res. 2022;12:1-16.
- [Google Scholar]
- Application of computer-aided diagnosis for Lung-RADS categorization in CT screening for lung cancer: effect on inter-reader agreement. Eur Radiol. 2022;32:1054-64.
- [CrossRef] [PubMed] [Google Scholar]
- Strategies for the management of solitary pulmonary nodules: A survey of patient preferences. Ann Thorac Surg. 2022;113:1670-5.
- [CrossRef] [PubMed] [Google Scholar]
- Risk stratification of indeterminate pulmonary nodules. Curr Opin Pulm Med. 2021;27:240-8.
- [CrossRef] [PubMed] [Google Scholar]
- Incidental pulmonary nodules-what do we know in 2022. Respiration. 2022;101:1024-34.
- [CrossRef] [PubMed] [Google Scholar]
- Management of pulmonary nodules by community pulmonologists: A multicenter observational study. Chest. 2015;148:1405-14.
- [CrossRef] [PubMed] [Google Scholar]
- Barriers to knowledge sharing in Chinese healthcare referral services: An emergent theoretical model. Glob Health Action. 2016;9:29964.
- [CrossRef] [PubMed] [Google Scholar]
- Factors associated with the inter-facility transfer of inpatients in Sichuan province, China. BMC Health Serv Res. 2019;19:329.
- [CrossRef] [PubMed] [Google Scholar]
- Factors influencing two-way referral between hospitals and the community in China: A system dynamics simulation model. Simulation. 2018;94:765-82.
- [CrossRef] [Google Scholar]