Translate this page into:
Effect of Picture Archiving and Communication System Image Manipulation on the Agreement of Chest Radiograph Interpretation in the Neonatal Intensive Care Unit
Address for correspondence: Dr. Donald Soboleski, Department of Diagnostic Radiology, Queen's University, Kingston, ON K7L 2V7, Canada. E-mail: daas@queensu.ca
-
Received: ,
Accepted: ,
This is an open access article distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License, which allows others to remix, tweak, and build upon the work non-commercially, as long as the author is credited and the new creations are licensed under the identical terms.
This article was originally published by Medknow Publications & Media Pvt Ltd and was migrated to Scientific Scholar after the change of Publisher.
Abstract
Objective:
Variability in image interpretation has been attributed to differences in the interpreters’ knowledge base, experience level, and access to the clinical scenario. Picture archiving and communication system (PACS) has allowed the user to manipulate the images while developing their impression of the radiograph. The aim of this study was to determine the agreement of chest radiograph (CXR) impressions among radiologists and neonatologists and help determine the effect of image manipulation with PACS on report impression.
Materials and Methods:
Prospective cohort study included 60 patients from the Neonatal Intensive Care Unit undergoing CXRs. Three radiologists and three neonatologists reviewed two consecutive frontal CXRs of each patient. Each physician was allowed manipulation of images as needed to provide a decision of “improved,” “unchanged,” or “disease progression” lung disease for each patient. Each physician repeated the process once more; this time, they were not allowed to individually manipulate the images, but an independent radiologist presets the image brightness and contrast to best optimize the CXR appearance. Percent agreement and opposing reporting views were calculated between all six physicians for each of the two methods (allowing and not allowing image manipulation).
Results:
One hundred percent agreement in image impression between all six observers was only seen in 5% of cases when allowing image manipulation; 100% agreement was seen in 13% of the cases when there was no manipulation of the images.
Conclusion:
Agreement in CXR interpretation is poor; the ability to manipulate the images on PACS results in a decrease in agreement in the interpretation of these studies. New methods to standardize image appearance and allow improved comparison with previous studies should be sought to improve clinician agreement in interpretation consistency and advance patient care.
Keywords
Agreement and consistency in interpretation
chest radiograph
manipulation of images
Neonatal Intensive Care Unit
picture archiving and communication system
INTRODUCTION
Chest radiographs (CXRs) are routinely utilized to access changes in status of newborns and infants in the neonatal intensive care setting and often lead to changes in management.[12] Differences in image interpretation have been attributed to multiple factors including differences in the interpreters’ knowledge base, experience level, and access to the clinical scenario. Inconsistent radiograph image quality due to patient movement, exposure setting alterations, and low lung volumes are a few of the many variables limiting assessment, and many researchers have demonstrated significant inter-observer variability in interpretation.[34567] The advent of picture archiving and communication system (PACS) has allowed the user to manipulate the images by altering the brightness (window level) and/or contrast (window width) of each radiograph while developing their impression. This manipulation will vary the appearance of the radiograph, and in effect, each user will be basing their impression on their own uniquely obtained image. This resulting image will be different from the image that their colleagues will generate to help them come to their impression. The purpose of this study is to determine the agreement of CXR impressions among radiologists and neonatologists and help determine the effect of image manipulation with PACS on reported diagnosis.
MATERIALS AND METHODS
The study was approved by the university and affiliated teaching hospitals research ethics board (REB), and patient informed consent was waived by the REB.
Patient population
This prospective cohort study included 60 nonconsecutive patients from the Neonatal Intensive Care Unit (NICU) (age range: 1 day to 3 months; 34 females, 26 males; gestational age range: 26–32 weeks) who had undergone CXRs as part of their routine care during a 3-month period in 2011. Patients included in the study had CXRs performed on variable days during their stay in the NICU. All patients had an underlying history of surfactant deficiency disease with follow-up radiographs ordered due to a concern of disease progression respiratory status. Exclusion criteria included infants who did not undergo two CXRs during their NICU stay. Images in which the CXR was combined with the abdomen were excluded from the study as well. The radiographs were obtained using 60 kVp and 1.5 mAs, the standard technique in the department.
Intervention
The sixty sets of frontal CXRs, each consisting of one previous and one recent examination, totaling 120 CXRs, were randomly placed as acquired into two identical viewers on PACS (GE Centricity 3.2, GE Healthcare IT, Barrington, Illinois, USA) used in the imaging department. The normally scanned requisition for the study was not included and each set of radiographs was sequentially numbered from1A/B to 60A/B. The previous or earlier study was placed to the right of the most recent study as per the department's usual routine. Three radiologists with a median length of expertise of 14 years (range: 5–25 years) along with three Neonatal Intensive Care Unit (NICU) physicians with a median length of expertise of 16 years (range: 2–25 years), each reviewed all 60 sets of frontal chest images in each of the two viewers. Each interpreter was instructed to provide a solitary impression of overall “improved,” “no change,” or “disease progression” lung disease for each of the 60 cases in each viewer. Each interpreter reviewed the 60 sets of cases in one viewer in their usual manner with manual windowing/manipulation as judged needed. Each interpreter (neonatologist or radiologist) reviewed the 60 sets of cases within the second viewer after an independent person (a radiologist) performed image windowing and contrast adjustment to best optimize the CXR appearance and to best match appearance with the previous study performed on that patient. The interpreters were not allowed to independently manipulate the appearance of the CXRs in this second viewer to come to their impression of whether the recent examination was improved, disease progression, or unchanged. Three interpreters began their review of the radiographs in the first viewer and the other three interpreters began with the second viewer. The time span between full review of the radiographs in one viewer and the completion of assessment of the radiographs in the other viewer ranged from 1 to 12 days.
Statistical analysis
Percent agreement was calculated between all six observers and separately for radiologists and neonatologists, for each of the two methods used in determining their impressions (allowing image manipulation/windowing and not allowing image manipulation/windowing). Similarly, the percent reporting opposing views, i. e., improved versus disease progression and vice versa, was calculated. Statistical significance was assessed using Chi-square test; the magnitude of the difference in agreement and opposition between manipulated and nonmanipulated impressions was assessed using relative risk (RR) and 95% confidence intervals (CI). The within observer kappa and the percent agreement, opposing, and unchanged impressions between manipulated and nonmanipulated methods were calculated. Statistical significance was set at <0.05. SPSS version 21 (IBM Corp. Released 2012. IBM SPSS Statistics for Windows, Version 21.0. Armonk, NY: IBM Corp.) and Microsoft Excel were used to conduct the analysis.
RESULTS
Overall, there was more agreement in CXR interpretations when images could not be manipulated. One hundred percent agreement in image impression (whether thought to be improved, disease progression or unchanged) between all six observers (three radiologists, three neonatologists) was only seen in 3 of the 60 cases (5%) when allowing image manipulation while 100% agreement was seen in 8 of the 60 cases (13%) when there was no manipulation of the images allowed (Chi-square = 2.5, P = 0.11) [Table 1]. Hence, the clinicians were 2.9 times more likely (95% CI 0.7, 14.8) to agree on image impressions when no manipulation of the images was permitted. There was 100% agreement between all three radiologists in 21 of the 60 cases (35%) when there was independent manipulation and 25 of the 60 cases (42%) without image manipulation (RR = 1.3, 95% CI 0.6–3.0). There was 100% agreement between all three NICU physicians in 8/60 (13%) of cases with independent manipulation and 18/60 (30%) of the cases when no manipulation was allowed (RR = 2.8, 95% CI 1.0–7.8). Overall, there were fewer opposing radiograph impressions when image manipulation was not allowed as compared to when manipulation/windowing was permitted [Table 1]. There were opposing image impressions (improved versus disease progression, vice versa) between all six interpreters in 37/60 (62%) of manipulated images and 29/60 (48%) of cases with no manipulation of images, resulting in a 40% decrease in opposing impressions when images could not be manipulated (RR = 0.6, 95% CI 0.3–1.3). Radiologists were less likely to report opposing impressions between the two methods compared to the neonatologists. Among the three radiologists, there were opposing impressions in 17/60 (28%) with manipulation and 12/60 (20%) with no manipulation of images (RR = 0.6, 95% CI 0.3–1.6). With the NICU physicians, there were opposing impressions in 24/60 (40%) with manipulation and 11/60 (18%) when no manipulation was allowed (RR = 0.3, 95% CI 0.1–0.8).
Radiologists were 3.5 times more likely than neonatologists to have 100% agreement on radiograph impressions when they had the ability to manipulate the image (95% CI 1.3–9.7) [Table 2]. Radiologists were also 1.7 times more likely to have 100% agreement on image interpretation when manipulation was not allowed (RR = 1.7, 95% CI 0.7–3.8) and 0.6 times less likely to have opposing impressions when manipulation was permitted (RR = 0.6, 95% CI 0.3–1.4). However, in some instances, the results were not statistically significant, primarily due to the small number of clinicians involved.
DISCUSSION
CXR interpretation is a routine essential requirement in the daily care of patients, particularly important in the NICU setting. Changes in the radiographic appearance result in a change in management in a significant percentage of patients.[8910111213] Multiple studies have demonstrated high inter-observer variability in the interpretation of radiographs. This has led to potential solutions such as double and triple reading of studies which are both costly and time-consuming.[67141516] More recent research has shown even less inter-observer agreement on digital compared to analog studies.[4] Numerous factors can contribute to poor image quality, such as patient and cassette positioning, exposure techniques, body habitus, and motion, and the radiographic images obtained are often suboptimal.[516] These variations in image quality and appearance often result in the interpreting clinician attempting to optimize the appearance by manipulating/windowing the image on PACS. The effect of this independent, individualized ability to manipulate the radiographic images has not been well documented.
Our study demonstrates an adverse effect in regards to the agreement in image interpretation when manipulation on PACS system is performed. The poorer inter-rater agreement was more significant among the NICU physicians, possibly due to less experience in image manipulation as compared to the radiologists who perform these maneuvers on a daily basis. The study also demonstrates overall poor agreement in image interpretation which has been shown in the medical literature.[367141516] Particularly, concerning is the percentage of opposing radiograph impressions which could result in significant changes to patient care and outcome. The ability to manipulate the appearance of the CXR images on PACS led to an increase in opposing interpretations in our study and raised the question of whether clinicians’ access to image windowing/manipulation is of benefit to the patient. Access to PACS and the ability to manipulate radiographic images independently by health care providers have increased in most hospital and patient care systems lately. This has been found to be very beneficial in regards to access data and patient radiographs; however, the potential disadvantages have not been elucidated. Further studies to demonstrate the impact on patient care and hospital costs are needed to help guide future endeavors by researchers, hospitals, and industry to improve our consistency in image interpretation. Novel and improved methods are needed to optimize image quality and allow for more reliable and consistent interpretation and comparison of radiographs.
Limitations
A pitfall of the study is the lack of a gold standard in the diagnosis of the radiograph abnormality. The study was designed to assess agreement between interpreters; therefore, accuracy per se was not deemed pertinent. All interpreters were specialists in their fields who routinely accessed the PACS as part of their daily routine of caring for patients. The inter-observer variability was thus regarded as a limitation of the test itself and not related to radiologist/neonatologist expertise. The dictating of the 60 sets of CXRs may have allowed for some patient recognition, and potential bias, although the lack of patient identification, the similar appearance of these types of studies and the alternating use of viewers are thought to have minimized this problem.
CONCLUSION
Agreement in CXR interpretation is poor, with conflicting impressions common with both neonatologists and radiologists. The ability to manipulate the images on PACS has resulted in a decrease in agreement in the interpretation of these studies. New methods to standardize image appearance and allow improved comparison with previous studies should be sought to improve clinician agreement in interpretation consistency and advance patient care.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
Acknowledgments
We would like to acknowledge Dr. J. Flood, M. Clarke, A. Acker and S. Salahudeen, who provided insight and expertise that greatly assisted our study. We would also like to acknowledge Wilma Hopman for her statistical and editorial expertise.
Available FREE in open access from: http://www.clinicalimagingscience.org/text.asp?2016/6/1/19/182730
REFERENCES
- Chest radiographs in 104 French ICUs: Current prescription strategies and clinical value (the RadioDay study) Intensive Care Med. 2012;38:1787-99.
- [Google Scholar]
- The clinical value of daily routine chest radiographs in a mixed medical-surgical intensive care unit is low. Crit Care. 2006;10:R11.
- [Google Scholar]
- Interobserver variation in interpreting chest radiographs for the diagnosis of acute respiratory distress syndrome. Am J Respir Crit Care Med. 2000;161:85-90.
- [Google Scholar]
- Interobserver variability in applying a radiographic definition for ARDS. Chest. 1999;116:1347-53.
- [Google Scholar]
- An assessment of inter-observer agreement and accuracy when reporting plain radiographs. Clin Radiol. 1997;52:235-8.
- [Google Scholar]
- Chest radiographic data acquisition and quality assurance in multicenter studies. Pediatr Radiol. 1997;27:880-7.
- [Google Scholar]
- Inter- and intra-observer variability in the assessment of atelectasis and consolidation in neonatal chest radiographs. Pediatr Radiol. 1999;29:459-62.
- [Google Scholar]
- Routine chest radiographs in pediatric intensive care: A prospective study. Pediatrics. 1989;83:465-70.
- [Google Scholar]
- Utility of routine chest radiographs in a medical-surgical intensive care unit: A quality assurance survey. Crit Care. 2001;5:271-5.
- [Google Scholar]
- Chest radiographs in surgical intensive care patients: A valuable “routine”. Henry Ford Hosp Med J. 1986;34:84-6.
- [Google Scholar]
- The value of routine daily chest x-rays in intubated patients in the medical intensive care unit. Crit Care Med. 1982;10:29-30.
- [Google Scholar]
- Efficacy of chest radiography in a respiratory intensive care unit. A prospective study. Chest. 1985;88:691-6.
- [Google Scholar]
- An integrated approach for prescribing fewer chest x-rays in the ICU. Ann Intensive Care. 2011;1:4.
- [Google Scholar]
- Routine daily chest radiography is not indicated for ventilated patients in a surgical ICU. Intensive Care Med. 1996;22:1335-8.
- [Google Scholar]
- Comparative diagnostic performances of auscultation, chest radiography, and lung ultrasonography in acute respiratory distress syndrome. Anesthesiology. 2004;100:9-15.
- [Google Scholar]
