View/Download PDF
ORIGINAL ARTICLE
2012
:2;
61
doi:
10.4103/2156-7514.102060
PDF

Assessing the Performance of Medical Personnel Involved in the Diagnostic Imaging Processes in Mulago Hospital, Kampala, Uganda

Ernest Cook Ultrasound Research and Education Institute (ECUREI), Kampala, Uganda
School of Public Health, Makerere University College of Health Sciences, Kampala, Uganda
School of Medicine, Makerere University College of Health Sciences, Kampala, Uganda
Address for correspondence: Dr. Michael G. Kawooya, ECUREI, C/O Mengo Hospital, Albert Cook Road, Albert Cook Building, P.O. BOX 7161 Kampala, Uganda. E-mail: kawooyagm@yahoo.co.uk
Licence
This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Disclaimer:
This article was originally published by Medknow Publications & Media Pvt Ltd and was migrated to Scientific Scholar after the change of Publisher of the Journal; therefore Scientific Scholar has no control over the quality or content of this article.
Source of Support: Nil
Conflict of Interest: None declared.

Abstract

Objectives:

Uganda, has limited health resources and improving performance of personnel involved in imaging is necessary for efficiency. The objectives of the study were to develop and pilot imaging user performance indices, document non-tangible aspects of performance, and propose ways of improving performance.

Materials and Methods:

This was a cross-sectional survey employing triangulation methodology, conducted in Mulago National Referral Hospital over a period of 3 years from 2005 to 2008. The qualitative study used in-depth interviews, focus group discussions, and self-administered questionnaires, to explore clinicians’ and radiologists’ performancerelated views.

Results:

The study came up with following indices: appropriate service utilization (ASU), appropriateness of clinician's nonimaging decisions (ANID), and clinical utilization of imaging results (CUI). The ASU, ANID, and CUI were: 94%, 80%, and 97%, respectively. The clinician's requisitioning validity was high (positive likelihood ratio of 10.6) contrasting with a poor validity for detecting those patients not needing imaging (negative likelihood ratio of 0.16). Some requisitions were inappropriate and some requisition and reports lacked detail, clarity, and precision.

Conclusion:

Clinicians perform well at imaging requisition-decisions but there are issues in imaging requisitioning and reporting that need to be addressed to improve performance.

Keywords

Medical imaging
performance
personnel

INTRODUCTION

Human resource for health

Uganda's ratio of radiologist to population is 1:2,500,000 and the number of imaging examinations per radiologist per year is 16,000.[1] Given this high workload, there is need to improve performance and efficiency.

Assessing physicians’ competence and performance

Physicians’ competence is defined as the routine and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.[2] Again competency has been defined as a complex set of behaviors built on knowledge, skills, and attitudes.[3] Based on this definition, it is evident that the commonly employed Licensure and board certification methods cannot effectively assess physician's competence or performance.[4]

Assessing physicians’ competence is important for purposes of improving performance of the physician, in addition to improving patient satisfaction.[56] Assessing of physicians’ performance is a complex undertaking requiring qualitative and quantitative evaluation.

Performance indicators are important for inculcating best practices and are linked to improved patient outcomes in healthcare, monitoring organizational health, and tracking progress toward institutional goals. Radiology-specific key performance indicators have been grouped as: operations management, financial management, patient safety, and quality of care, those relating to external and internal stake holders.[7]

Varying departmental performance indicators have been suggested for evaluating, organization, volume and productivity, radiology reporting, customer satisfaction, and finance among others. A study carried in the US, showed that many academic radiology departments do not use indicators and there was no agreement as to which indicators to use. Most commonly used indicators aimed at monitoring productivity, especially through measurement of examination volumes. Those departments, which measured productivity, coupled this to financial indicators.[8]

Operational definition of performance indices

The “user” for purposes of this study is the referring clinician and the radiologist. Appropriate service utilization (ASU) is the proportion of patients for whom the decision by the clinician to requisition for imaging is appropriate. Appropriateness of the clinician's nonimaging decision (ANID) is the proportion of patients for whom the “decision by the clinician, not to requisition” is appropriate. The clinical utilization of imaging (CUI) is the proportion of imaging findings, utilized for patient management, out of all patients who undergo imaging.

OBJECTIVES

The first objective was to develop and apply three imaging performance indices namely, ASU, ANID, and CUI. The second was to assess the validity of the referring clinician in imaging-requisition decision-making. Others were: documenting the nontangible aspects of user performance and eliciting suggestions toward performance improvement.

MATERIALS AND METHODS

Study methods

Study design

This was a cross-sectional survey with triangulation. For the quantitative part of the study, cluster sampling was applied. The clusters were obstetrics and gynecology (OB/GYN), surgery, internal medicine, and pediatrics. The qualitative component employed purposive sampling.

Study areas

The study site was Mulago, Uganda's main tertiary hospital, which has a capacity of 2500 beds.

Sample size and sampling procedure for the quantitative component of the study

Sample size was estimated by the Kish and Leslie formula. Cluster sampling was employed. The study sample consisted of 384 patients divided into four clusters.[9] Systematic sampling was applied recruiting every 5th patient within a 3 months study period.

Methods of data collection for the quantitative component of the study

The data pertaining to imaging was extracted from the patients’ case notes. This information was recorded on precoded data sheets and used for rating for appropriateness of the imaging and nonimaging decisions and subsequently for calculating the performance indices.

Rating for appropriateness of imaging and nonimaging decisions and for clinical utilization of imaging results

A group of three peer raters excluding the principal investigator (PI) rated each patient's case-information, as to whether the imaging decision or the decision not to image was appropriate. The raters also rated the case notes as to whether the results of imaging had an impact on subsequent patient management. Each case note was initially rated by two raters. Rating was independent and each rater was blinded to the score of the other raters. The rating was based on a set of previously agreed on criterion designed by the raters together with the PI for purposes of this study. If the two raters agreed, there was no need for a third rater, but if they disagreed, then the third rater came in as a tie breaker. This information was used to calculate the four needs indices.

Data collection for the qualitative part of the study

This study component probed deficiencies in imaging requisitions and imaging reports and how these could be rectified. Twenty-two in-depth interviews (IDI), 4 focus group discussions (FGDs), and 42 self-administered questionnaires (SAQ) were employed. These were administered to clinicians and radiologists.

Calculation of indices

Appropriate service utilization

The denominator was all imaging requests written for patients in a given hospital within a specified period, and the numerator, the appropriate requests in that same hospital and period.

Appropriateness of nonimaging decisions

The denominator were the patients seen during the study period, that did not deserve to be imaged and the numerator were those patients, for whom it was deemed correct by the clinician not-to-requisition for imaging.

Clinical utilization

The denominator was all imaging results obtained and the numerator was those results rated by the PI as having been utilized for patient management.

This followed the method for calculating sensitivity, specificity, negative, and predictive values using a 2 × 2 table [Table 1]. The result of the three raters was assumed to be the “gold standard”. Identification of those patients needing imaging by the clinician (appropriate requisition) is equated to the sensitivity of a test and from the 2 × 2 table, this is:

Table 1: Sensitivity and specificity of the clinician for appropriate requisitioning

where, A = True positives, C = False negatives

Identification of those not needing imaging by the clinician is equivalent to specificity:

where, D = True negatives, B = False positives

The predictive values and likelihood ratios were similarly calculated.

Potential sources of bias in calculation of indices

The two possible causes of bias in this study were: inability to accurately define the outcome variable (namely appropriate and nonappropriate requisitions) and inability to get a gold test or gold standard for appropriateness. As a solution, a criterion for appropriateness was developed and applied. A third rater was brought in as a tie-breaker in case the two raters disagreed.

Ethical clearance

Ethical clearance was obtained from the Uganda National Council for Science and Technology.

RESULTS

The appropriate service utilization

The ASU was 94% and was based on a sample size of 353 patients whose age ranged from 1 to 90 years, with a mean of 22.3 years, standard deviation of 20.5 and a male to female ratio of 1:1. The ASU was highest (100%) among the OB/GYN, followed by the pediatric cluster (97%). It was least for surgery (89%) and internal medicine (83%) [Figure 1]. It was highest (100%) for computed tomography (CT) examination, followed by ultrasound (98%). It was lowest for conventional radiography (93%) [Figure 2]. Senior clinicians (consultants) scored a higher ASU of 97%, compared with juniors (89%) (Pearson's Chi square = 0.197).

Figure 1: Appropriate service utilization (ASU) within clusters. ASU was highest for internal medicine cluster, followed by the pediatrics cluster. The surgery cluster scored lowest.
Figure 2: Appropriate service utilization (ASU) for types of imaging investigations. ASU was highest for the more sophisticated imaging investigations namely contrasted X-ray and CT/MRI, and lowest general X-rays.

The appropriate nonimaging decision

The ANID was 80.0% and was based on a sample size of 301 patients whose age ranged from 1 to 85 years. It was highest in the OB/GYN (86%), followed by pediatrics (85%). It was lowest in the internal medicine and surgery clusters (71%).

The clinical utilization of imaging

The overall CUI was 97% and was based on a sample size of 202 patients whose age ranged from 1 to 79 years. It was highest (100%) in the OB/GYN and pediatric clusters and lowest in the surgery (94%) and internal medicine (93%) clusters [Figure 3]. Ultrasound and CT scored best (100%) among imaging techniques and conventional radiography scored least (94%) [Figure 4]. Normal imaging results were utilized in 96% of cases and abnormal in 97%. The imaging results led to change in treatment in 68% of patients, modification of treatment in 25% and requisition of additional imaging modalities in 6% of the patients.

Figure 3: Clinical utilization of imaging results (CUI) within clusters. CUI was highest in the obstetrics and gynecology, followed by surgery. It was lowest in the pediatrics cluster.
Figure 4: Clinical utilization of imaging results (CUI) and types of imaging. CUI was highest for contrasted studies followed by ultrasound and lowest for CT/MRI.

The validity of the clinician in identifying correctly those patients who need imaging (sensitivity) and those who do not (specificity)

Correct identification by the clinician, of patients who needed imaging is equated to the sensitivity of a test and was 85%, whereas specificity was 92%. The positive predictive value was 94% and negative predictive value 80%. The positive likelihood ratio was 10.6 and negative likelihood ratio 0.16.

The qualitative component of the study

Problems with imaging requisitioning

The problems were grouped under the following categories: incomplete requisitions, vague requisitions, anonymous requisitions, and requisitions, which were inappropriate in that they were not relevant to the patients’ illness. With regard to incomplete requisitions, the rationale and urgency for imaging were not always explicit. Key demographic information was often missing. On vague requisitions, one clinician expressed: “so if our requisitions are imprecise, how do we expect the radiologist to give a report focused on what we want?”

Causes of poor requisitions were expressed by interviewees using these phrases: “improper clinical work up”, the “short in the dark” approach and the “let's do something as we wait” approach.

Ways to improve requisitions

The interviewees proposed ways of improvement are categorized as follows: care while conducting physical examination of the patients, care in writing requisitions, ensuring appropriate choice of the examination, and better support supervision for junior doctors.

Deficiencies and problems with imaging reports

The deficiencies were grouped under categories: varying report styles, unfamiliar terminology, brief descriptive section, discrepancies within imaging reports, inconclusive reports, unfocussed differential diagnosis, and irrelevant recommendations.

Within the category of varying report styles were lack of uniformity and unfamiliar terminology. One said: “There are some terms that you use, unfortunately, we cannot understand them, like echogenicity”. Unfamiliar terminology was with the ultrasound, CT, and magnetic resonance imaging (MRI). This issue raised passionate arguments with some radiologists advocating for use of terminology, especially in the descriptive section of the report. One radiologist said: “I think there is a discrepancy between the expectations of the clinicians and what the radiologist provides in the report. The radiologist cannot make an exhaustive description without using radiology words”.

There was a complaint over brief descriptive section. One criticized: “so, if the report is very short and it just jumps to conclusions, you start asking yourself how one came to those conclusions without describing exactly what they found”.

Discrepancies with clinico-laboratory data were pointed. Some doctors complained that some reports were inconclusive, hence, as one stated: “leaving me in limbo as to what the problem and appropriate intervention could be”.

Views on improving reports related to user competence

Views were advanced by clinicians on how to improve reports. These are categorized as: use of familiar language and correct grammar, clear and uniform reporting style, detailed description, conclusive reports, focused differential diagnosis, care while giving recommendations, and stating limitations of the examinations.

Radiologists and clinicians agreed that some terminologies were unavoidable in the descriptive section but should be avoided in the conclusion. An uniform reporting style was recommended. The descriptive section of the report should be detailed, clear, and tally with the clinical picture.

There were contrasting views on the conclusive part of the report with some in favor of a conclusion and others were not. It was suggested that the differential diagnosis be more focused and preferably include not more than three diagnoses.

The section on recommendations was a subject of much discussion. One clinician raised the eyebrows of the radiologists participating in the FGD by stating: “Sometimes I act on them, sometimes I do not; it depends on what extra information I need, so often I do not always follow them”. Another responded: “if there are recommendations to be made, they should be made as suggestions, but not strong recommendations”. One surgeon ushered in calm by advising that: “Such recommendations are quite useful and we as surgeons respect them and they guide us”.

DISCUSSION

Appropriateness of imaging decisions

The overall ASU of 94% obtained in this study is higher than that of Kahn et al.,[10] who recorded appropriateness levels of up to 78% among imaging requisitions from a general internal medicine clinic in the USA.

The high ASU is an indication that the clinician fairly accurately predicted those patients who would benefit from imaging and this may be due to pressure from limited resources. Consequently, more expensive investigations like CT, MRI, and fluoroscopy carry a higher ASU. A relatively low ASU for the surgical cluster may be attributable to inappropriate requisitions for trauma cases. This calls for appropriateness criteria to improve on the ASU.

The higher ASU for the more senior clinician illustrates the importance of clinical aptitude and experience in making appropriate requisition decisions though this is not always the case.[11]

Requisitioning and report-writing guidelines may improve performance but their effect is not uniform.[12]

Appropriateness of non-imaging decisions

The overall ANID for Mulago, of 80%, may be explained by the caution the clinician exercises while requisitioning for imaging in the light of limited imaging resources. The lower ANID for surgery may be because many imaging studies are ordered in the emergency room and this is usually by junior doctors. Assessment of the ANID is important in making sure that all patients who would benefit from imaging receive the service and in monitoring inappropriate requisitions.[13]

The clinical utilization of imaging

A CUI of 97% may reflect the performance of the referring clinician and the radiologist. It is interesting to note that both negative and positive imaging results impact patient management. The high CUI is evidence that the radiologists take care to issue an accurate report and that the clinician utilize the results for subsequent patient management.

The CUI may reflect how the subsequent management intervention is solely dependent on the imaging finding hence it is higher for OB/GYN and pediatrics. It is interesting that ultrasound and CT had the highest CUI. It is possible that in these particular clinical scenarios, ultrasound had a high utility.[14]

The validity of the clinician as a measuring tool, in deciding which patients would benefit from imaging

The positive likelihood ratio of 10.6 and negative ratio 0.16 show that the clinician is a good measuring tool for identifying patients who may benefit from imaging in comparison to those who will not.

Problems with imaging requisitioning and proposed solutions

The main problems identified and the causes are similar to those by other researchers.[1516] Solutions like enhancing interactions between radiologists and clinicians plus provision of imaging guidelines have been proposed by others.[1719]

Deficiencies with imaging report and how to rectify them

The study revealed areas of deficiencies in imaging reports, which included incomprehensive terminologies, nonauthoritative reports, and inconclusive reports. The clinicians suggested improvements. Other authors have advocated for promotion of professionalism and communication skills for radiologists so as to improve patient satisfaction and outcomes.[5]

In this study, ultrasound, CT, and MRI were found to have the most incomprehensive terminologies. Other authors noted variability of readability indices with different procedure.[20]

This study suggests limiting terminology to the descriptive section, and using ordinary language in the conclusion. Use of proper descriptive terms has been recommended by others.[2122] This study, like others reveals that clinician want more descriptive detail.[2123] Clinicians asked for more authoritative reports, an observation also made by other workers.[20] This study findings highlight the importance of the conclusion, and other authors concur with this observation.[21] The importance of limited and pertinent differential diagnoses has been underscored by others.[21] This contention and debate on recommendations is not new in literature.[2324] Recommendations should be to the patient's interest, not overzealous, and with a clinical basis.[17]

CONCLUSION

Three indices for measuring the performance of medical personnel involved in imaging have been developed and applied. Nonmeasurable aspects of user performance like quality of imaging requisitions and reports need attention.

REFERENCES

  1. ASR. Numbers of imaging personnel in African countries. Archives of African Society of Radiology (ASR)
  2. , , . Defining and assessingprofessional competence. JAMA. 2002;287:226-35
    [Google Scholar]
  3. , , , , . Educating the pediatrician of the 21st century: Defining and implementing a competencybased system. Pediatrics. 2004;113:252-8
    [Google Scholar]
  4. , , , , . Physician Clinical Performance Assessment Prospects and Barriers. JAMA. 2003;290:1183-9
    [Google Scholar]
  5. , , . Establishing a program to promote professionalism and effective communication in radiology radiology. Radiology. 2006;238:773-9
    [Google Scholar]
  6. , , , , , , . What health professionals can do to identify and resolve patient dissatisfaction. Jt Comm J Qual Improv. 1998;24:303-12
    [Google Scholar]
  7. , , , , . Key performance indicators for measuring and improving radiology department performance. Radiographics. 2010;30:571-83
    [Google Scholar]
  8. , , , , , , . Practice management, performance indicators, in academic radiology departments. Radiology. 2004;233:716-22
    [Google Scholar]
  9. , . C sample: Analysing data from complex survey samples. Epi info version 6, users guide. . 2000;157:82
    [Google Scholar]
  10. , , , , , , . Appropriateness of imaging procedure requests: Do radiologists agree? AJR Am J Roentgenol. 1997;169:11-4
    [Google Scholar]
  11. , . Skull radiography in the evaluation of acute head trauma: A survey of current practice. Radiology. 1991;181:711-4
    [Google Scholar]
  12. Royal College of Radiologists Working Party, 1992, Influence of the Royal College of Radiologists’ guidelines on hospital practice: A multicentre study. BMJ. 1992;304:740-3
    [Google Scholar]
  13. , , , , , , . Measuring the necessity of medical procedures. Med Care. 2011;32:357-65
    [Google Scholar]
  14. , . Utility assessment and clinical trains of diagnostic interventions. Acad Radiol. 1999;6(Suppl 1):S103-8
    [Google Scholar]
  15. , . Sounding board: The overutilization of x-rays. N Engl J Med. 1979;300:1213-6
    [Google Scholar]
  16. , . Overutilization of radiological examinations. Radiology. 1976;120:443-8
    [Google Scholar]
  17. , , , , , . The contemporary radiologist: Consultant or film reader? AJR Am J Roentgenol. 1997;169:353-4
    [Google Scholar]
  18. , , . A recommendation on recommendations. AJR Am J Roentgenol. 1997;169:351-2
    [Google Scholar]
  19. , , , , . Requesting diagnostic imaging examinations: A position paper of the Canadian Association of Radiologists. Can Assoc Radiol J. 1997;48:89-91
    [Google Scholar]
  20. , . Standardized mammography reporting. Radiol Clin North Am. 1992;30:257-64
    [Google Scholar]
  21. , , . Malpractice issues in radiology. American College of Radiology standards for communication. AJR Am J Roentgenol. 1999;173:1432-42
    [Google Scholar]
  22. Evidence-based radiology, a new approach to the practice of radiology. Radiology. 2001;220:566-75
    [Google Scholar]
  23. , , , . Style guidelines for radiology reporting: A manner of speaking. AJR Am J Roentgenol. 2003;180:327-8
    [Google Scholar]
  24. , . Language of the radiology report. Primer for residents and wayward radiologists. AJR Am J Roentgenol. 2000;175:1239-42
    [Google Scholar]
Show Sections