Performance of Radiologists in the Evaluation of the Chest Radiography with the Use of a “new software score” in Coronavirus Disease 2019 Pneumonia Suspected Patients

INTRODUCTION

The severe acute respiratory syndrome coronavirus 2 is a newly emerging zoonotic agent appeared in December 2019 and causing the coronavirus disease 2019 (COVID-19).^[1] The virus rapidly infected all provinces of China and spread to the rest of the world.^[2] COVID-19-infected patients develop pneumonia with associated symptoms of fever (98%), cough (76%), and myalgia or fatigue (44%).^[3] The infection results in a syndrome leading, in some cases, to a critical care respiratory condition, requiring specialized management at intensive care units.^[4,5] The current diagnostic criterion for COVID-19 is the positive result of a nucleic acid test by real-time reverse transcription-polymerase chain reaction (RT-PCR).^[2]

Thoracic imaging with chest radiography (or chest X-ray, CXR) and computed tomography (CT) plays an important role for diagnosis, management, and follow-up of COVID-19 pneumonia patients.^[6] The typical radiological imaging demonstrated clear destruction of the pulmonary parenchyma, including interstitial inflammation and extensive consolidation.^[7] Many groups describe lung opacities in more than one lobe and bilateral lower lobe consolidations (as opposed to community acquired bacterial pneumonia which tends to be unilateral and involving a single lobe). Identifying multifocal air-space disease on CXR can be a significant clue to COVID-19 pneumonia;^[8] some investigators have noticed that the air-space disease tends to have a lower bilateral lung distribution (in the 33–85% of cases).^[8,9] High frequency of peripheral lung involvement (in the 77–88% of cases) is another feature readily identifiable on CXR. Ground glass opacities emerges in other cases evaluated (GGO, in the 55–79% of cases), and it is extremely difficult to detect on CXR.

CXR is a less sensitive modality in the detection of COVID-19 lung disease compared to CT (with a reported baseline CXR sensitivity of 69%).^[9] CT is more sensitive than CXR for early disease and it is also important to exclude alternative diagnoses (pulmonary thromboembolism, and heart failure),^[10,11] but the use of CXR could offer some advantages. However, due to infection control issues related to patient transport to CT suites, the inefficiencies introduced in CT room decontamination, and lack of CT availability in parts of the world, portable chest radiography (CXR) will likely be the most commonly utilized modality for identification and follow-up of lung abnormalities. In fact, the American College of Radiology notes that CT decontamination required after scanning COVID-19 patients may disrupt radiological service availability and suggests that portable chest radiography may be considered to minimize the risk of cross-infection.^[12] Furthermore, in cases of high clinical suspicion for COVID-19, a positive CXR may obviate the need for CT.

Many groups have tried to schematize radiologist’s work providing various “diagnostic scores” with the aim to make the report more standardized; actually, all of these researches are “CT score.”

The use of a specific score for CXR report has not been well investigated; the purpose of this study is to clarify the use of a specific CXR digital tool and to investigate its validity in terms of inter-reader concordance.

MATERIAL AND METHODS

Case selection and imaging evaluation

From February 24, 2020, to April 3, 2020, more than 1560 patients with suspected diagnosis of COVID-19 viral pneumonia underwent chest radiography (CXR) in our institution. To evaluate a new tool called “COVID-19 score” made available to radiologists for lung imaging analysis, we retrospectively included in the study patients who underwent at least two consecutive chest X-rays for a total of 312 exams.

This instrument is a software function available within the radiological examination reporting platform of our institute; a banner depicting a lung image is opened by a click of the mouse, as shown in Figure 1. The radiologists must assign a score from 0 to 3 for to the various lung areas of each hemothorax (a: Upper; b: Intermediate; c: Lower on the right; and d: upper; e: Intermediate, and f: Lower on the left). The upper area includes from the apices to the profile of the aortic arch (Line A); the intermediate zone extends from the hilar structures (from Line A) to the lower margin of the right lower pulmonary vein (Line B) and the lower area extending from Line B to the diaphragm.

Close

Figure 1:

The software banner depicting a lung image to obtain the CXR “COVID-19 score.” This picture shows the three different areas of each lung (a-c-b for the left and d-e-f for the right), the upper area (from the apices to the profile of the aortic arch – Line A), the intermediate area (from the Line A to the lower edge of the inferior pulmonary – Line B), and the lower (from the Line B to the diaphragm).

Export to PPT

For each zone radiologists assigned the score based on predominant features, graded on a 3-point scale as follow: (0) For negative exam; (1) for predominant interstitial changes (as reticular and/or interlobular septal thickening); (2) in the presence of both interstitial changes and alveolar consolidation (as interstitial changes predominance); and (3) in the presence of both interstitial changes and alveolar consolidation (as alveolar consolidation predominance).

Points from the values for each zone (from A to F) were added to obtain a final total cumulative score (from 0 to 18). The software can automatically generate the final “COVID-19 score;” the data can be recorded and filed for display in subsequent reports and it can be used as a comparison parameter in sequential radiological examinations (for a temporal evaluation of the trend).

The radiologists retrospectively and blindly assessed the pool of selected chest X-ray exams and reported the “COVID-19 score” values in an Excel database. When filing, the radiologists had no clinical information and did not know the outcome of the RT-PCR COVID test (actually considered the gold standard for the diagnosis). Chest X-ray images were viewed on Barco workstation.

The time required to complete the banner of each exam for the achievement of the overall score was reported and the insertion of the “COVID-19 score” within the structured reports of CXR was assessed, as a prognostic evaluation parameter.

The practical implications of using this new tool in the radiological CXR evaluation were considered.

RESULTS

Fourteen radiologists (three of the Group 1; five of the Group 2; three of the Group 3; and three of the Group 4) evaluated 312 chest X-ray images of COVID-19 pneumonia suspected patients (80 males and 38 females) with an average age of 64, 47 years.

The radiologists belong to the three hospitals of our company (one from Ivrea, nine from Ciriè, and four from Chivasso). No radiologist has reported difficulties in accessing the software function or in filling out the banner. No issues with the automatic registration of the “COVID-19 score” were found.

The results of the inter-rater and intra-group concordance, obtained with the Fleiss’ Kappa statistic, showed a moderate agreement (0.53) among the radiologists (considering ranges: K <0.4 for a concordance definable as “poor;” 0.4 < K < 0.6 definable as “moderate;” 0.6 < K < 0.8 definable as “good;” and K > 0.8 definable as “excellent”). The intra-group analysis leads to the same conclusion, with Fleiss’ Kappa values between 0.49 for Group 1 and 0.59 for Group 3, indicating that the years of work experience was irrelevant in the use of this new software application.

The average time for filling in the banner of a chest X-ray for obtaining the “COVID-19 score” with automatic software was: Seven seconds for negative exams (with score equal to zero); 13 s for tests with score between one and six; 21 s for exams with intermediate score (from 7 to 12, where the presence of more pathological findings made it more difficult to identify and interpret the image), and 15 s for images strongly indicative of pulmonary impairment (with score values from 13 to 18).

DISCUSSION

Rapid and accurate diagnosis of COVID-19 is critical during the epidemic.

Thoracic imaging with chest radiography (CXR) and CT is key tools for pulmonary disease diagnosis and management, but their role in the management of COVID-19 has not been considered within the multivariable context of the severity of respiratory disease, pre-test probability, risk factors for disease progression, and critical resource constraints.^[13,14]

To solve these problems, we hypothesize a diagnostic model based on radiological imaging and clinical manifestations alone, independent of the nucleic acid test.

Analysis of specific cases, where radiologists were wrong reveals that the mistakes were made when the COVID-19 chest imaging findings are either subtle (likely reflecting early time in the disease process) or when COVID-19 has atypical signs. It is worth noting that non-COVID-19 pneumonia can also have typical appearance of COVID-19. This poses a dilemma because mandated quarantine for all suspected cases can put significant strain on medical infrastructure, health-care providers, and the lives of patients, but may need to be followed as a necessary precaution due to variation in presentation with timing of disease.^[15]

Multiple studies demonstrated that the common COVID-19 chest imaging findings are represented by bilateral involvement, peripheral distribution, and lower zone dominance; these can also be appreciated on CXR.^[8] In the scenario, where there is high clinical suspicion of COVID-19 it is conceivable that a positive CXR may obviate the need for a CT, thus reducing burden on CT units in this pandemic.

Although chest CT has high sensitivity, it has low specificity.^[10,16] This low specificity may stem from the fact that it is difficult to distinguish COVID-19 findings from findings of other disease on chest CT. Another critical issues should be considered: The availability of the CT scanner and the time required to clean and disinfect the equipment after performing imaging on a suspected patient.

This problem can be solved using CXR portable equipment. The use of these mobile devices allows the exam to be performed in the same room the patient is isolated into, therefore reducing the risk of COVID-19 transmission along the transport route to a CT scanner. In hospitalized patients, CXR can be used as a first step to assess the progression of the disease or to make alternative diagnoses, such as lobar pneumonia (suggestive of bacterial superinfection), pleural effusion, or pneumothorax.

The role of imaging in COVID-19 patients’ management should be considered in a multivariable context; the physicians should try to choose the best tools in a specific context considering different practice settings, different phases of the epidemic outbreak, and specific resources availability.

In this study, we tested a new software application called “COVID-19 score” that can be used in the reporting of chest X-ray imaging in patients suspected COVID-19, based on radiological semantic features. Using this tool, we found some critical issues: The range (proposed by the design company) with values from 0 to 18 is too wide (it does not allow a precise statistical analysis and generates multiple interpretative variables in radiologists). In addition, the “COVID-19 score” obtained with the cumulative method (by automatic addition and without a specific attribution of the area/s concerned) is difficult to correlate with the true clinical status of the patient.

By stratifying the scores into three classes (0–4, 5–13, and 14–18), we saw that the extreme values can only be representative of subject’s symptomatology. In fact, the class with “COVID-19 score” from 0 to 4 correlates with negative imaging and pauci-symptomatic patient. The class with “COVID-19 score” from 15 to 18 correlates with pathological imaging and with severely symptomatic patient is shown in Figure 2. On the other hand, the intermediate class with “COVID-19 score” from 5 to 13, poses major interpretative problems, due to the copresence of different situations even with identical score values, as shown in Figures 3 and 4. For example, a score of six could derive both from the sum of a score of one in all anatomical areas (from A to F) and from the sum of a maximum score attributed in two ipsilateral anatomical areas (equivalent to a 2/3 lung pathological involvement) or contralateral (equivalent to a bilateral pathological involvement process).

Close

Figure 2:

(a) 69-year-old man with pneumonia disease who presented with fever, cough, and dyspnea. Portable CXR image shows lungs with bilateral pulmonary involvement with widespread interstitial disease and areas of consolidation in different zones of each lung with CXR “COVID-19 score” value of 18; (b) 64-year- old man with respiratory distress (intubated and subjected to mechanical ventilation). Portable CXR image shows lungs with multiple small areas of lung consolidation affecting the whole lung parenchyma bilaterally with CXR “COVID-19 score” value of 18.

Export to PPT

Close

Figure 3:

65-year-old man arrived in the emergency room with sudden high fever (39.8°C) and intense cough. (a) The first portable CXR examination shows lungs with minimal disventilative bibasal areas associated with reticular and hazy medium and the lower left lobe opacities (with interstitial changes predominance) with CXR “COVID-19 score” value of 3. (b) After a week, the second portable CXR exam shows lungs with progressive worsening with bilateral medium and lower lobe consolidations (with alveolar consolidation predominance) with CXR “COVID-19 score” value of 8.

Export to PPT

Close

Figure 4:

(a) 67-year-old man with respiratory distress (intubated and subjected to mechanical ventilation). Portable CXR image shows lungs with multiple small areas of lung consolidation (with alveolar consolidation predominance) in the whole lung parenchyma bilaterally with CXR “COVID-19 score” value of 16. (b) 3-years-old man with respiratory distress (intubated and subjected to mechanical ventilation). Portable CXR image shows lungs with reticular and hazy medium and lower bilateral lobe opacities (with interstitial changes predominance) with CXR “COVID-19 score” value of 12.

Export to PPT

To overcome this correlation problem between “score imaging” and “clinic status,” we propose that the “COVID-19 score” in CXR be weighed on the number of anatomical lung sectors (A- B) involved; this could offer clinicians a quantifiable data that can be used for patient monitoring over time.

CONCLUSION

The use of automatic software for the generation of a “COVID-19 score” within the reports of chest X-rays in COVID-19 pneumonia suspected patients has proven to be simple, fast, and replicable in different working contexts with a moderate agreement among the radiologists and without significant differences related to the work experience. Implementing this tool with scores weighed on the number of lung pathological areas can be a future parameter of clinical interest for the prognostic temporal evaluation of the improvement or worsening trend of a patient’s health.