Assessing the Performance of Medical Personnel Involved in the Diagnostic Imaging Processes in Mulago Hospital, Kampala, Uganda

Human resource for health

Uganda's ratio of radiologist to population is 1:2,500,000 and the number of imaging examinations per radiologist per year is 16,000.[1] Given this high workload, there is need to improve performance and efficiency.

Assessing physicians’ competence and performance

Physicians’ competence is defined as the routine and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.[2] Again competency has been defined as a complex set of behaviors built on knowledge, skills, and attitudes.[3] Based on this definition, it is evident that the commonly employed Licensure and board certification methods cannot effectively assess physician's competence or performance.[4]

Assessing physicians’ competence is important for purposes of improving performance of the physician, in addition to improving patient satisfaction.[56] Assessing of physicians’ performance is a complex undertaking requiring qualitative and quantitative evaluation.

Performance indicators are important for inculcating best practices and are linked to improved patient outcomes in healthcare, monitoring organizational health, and tracking progress toward institutional goals. Radiology-specific key performance indicators have been grouped as: operations management, financial management, patient safety, and quality of care, those relating to external and internal stake holders.[7]

Varying departmental performance indicators have been suggested for evaluating, organization, volume and productivity, radiology reporting, customer satisfaction, and finance among others. A study carried in the US, showed that many academic radiology departments do not use indicators and there was no agreement as to which indicators to use. Most commonly used indicators aimed at monitoring productivity, especially through measurement of examination volumes. Those departments, which measured productivity, coupled this to financial indicators.[8]

Operational definition of performance indices

The “user” for purposes of this study is the referring clinician and the radiologist. Appropriate service utilization (ASU) is the proportion of patients for whom the decision by the clinician to requisition for imaging is appropriate. Appropriateness of the clinician's nonimaging decision (ANID) is the proportion of patients for whom the “decision by the clinician, not to requisition” is appropriate. The clinical utilization of imaging (CUI) is the proportion of imaging findings, utilized for patient management, out of all patients who undergo imaging.

Study methods

Sample size and sampling procedure for the quantitative component of the study

Sample size was estimated by the Kish and Leslie formula. Cluster sampling was employed. The study sample consisted of 384 patients divided into four clusters.[9] Systematic sampling was applied recruiting every 5^th patient within a 3 months study period.

Methods of data collection for the quantitative component of the study

The data pertaining to imaging was extracted from the patients’ case notes. This information was recorded on precoded data sheets and used for rating for appropriateness of the imaging and nonimaging decisions and subsequently for calculating the performance indices.

Rating for appropriateness of imaging and nonimaging decisions and for clinical utilization of imaging results

A group of three peer raters excluding the principal investigator (PI) rated each patient's case-information, as to whether the imaging decision or the decision not to image was appropriate. The raters also rated the case notes as to whether the results of imaging had an impact on subsequent patient management. Each case note was initially rated by two raters. Rating was independent and each rater was blinded to the score of the other raters. The rating was based on a set of previously agreed on criterion designed by the raters together with the PI for purposes of this study. If the two raters agreed, there was no need for a third rater, but if they disagreed, then the third rater came in as a tie breaker. This information was used to calculate the four needs indices.

Data collection for the qualitative part of the study

This study component probed deficiencies in imaging requisitions and imaging reports and how these could be rectified. Twenty-two in-depth interviews (IDI), 4 focus group discussions (FGDs), and 42 self-administered questionnaires (SAQ) were employed. These were administered to clinicians and radiologists.

Calculation of indices

Clinical utilization

The denominator was all imaging results obtained and the numerator was those results rated by the PI as having been utilized for patient management.

This followed the method for calculating sensitivity, specificity, negative, and predictive values using a 2 × 2 table [Table 1]. The result of the three raters was assumed to be the “gold standard”. Identification of those patients needing imaging by the clinician (appropriate requisition) is equated to the sensitivity of a test and from the 2 × 2 table, this is:

Table 1 Sensitivity and specificity of the clinician for appropriate requisitioning

where, A = True positives, C = False negatives

Identification of those not needing imaging by the clinician is equivalent to specificity:

where, D = True negatives, B = False positives

The predictive values and likelihood ratios were similarly calculated.

Potential sources of bias in calculation of indices

The two possible causes of bias in this study were: inability to accurately define the outcome variable (namely appropriate and nonappropriate requisitions) and inability to get a gold test or gold standard for appropriateness. As a solution, a criterion for appropriateness was developed and applied. A third rater was brought in as a tie-breaker in case the two raters disagreed.

Ethical clearance

Ethical clearance was obtained from the Uganda National Council for Science and Technology.

The appropriate service utilization

The ASU was 94% and was based on a sample size of 353 patients whose age ranged from 1 to 90 years, with a mean of 22.3 years, standard deviation of 20.5 and a male to female ratio of 1:1. The ASU was highest (100%) among the OB/GYN, followed by the pediatric cluster (97%). It was least for surgery (89%) and internal medicine (83%) [Figure 1]. It was highest (100%) for computed tomography (CT) examination, followed by ultrasound (98%). It was lowest for conventional radiography (93%) [Figure 2]. Senior clinicians (consultants) scored a higher ASU of 97%, compared with juniors (89%) (Pearson's Chi square = 0.197).

Close

Figure 1

Appropriate service utilization (ASU) within clusters. ASU was highest for internal medicine cluster, followed by the pediatrics cluster. The surgery cluster scored lowest.

Export to PPT

Close

Figure 2

Appropriate service utilization (ASU) for types of imaging investigations. ASU was highest for the more sophisticated imaging investigations namely contrasted X-ray and CT/MRI, and lowest general X-rays.

Export to PPT

The appropriate nonimaging decision

The ANID was 80.0% and was based on a sample size of 301 patients whose age ranged from 1 to 85 years. It was highest in the OB/GYN (86%), followed by pediatrics (85%). It was lowest in the internal medicine and surgery clusters (71%).

The clinical utilization of imaging

The overall CUI was 97% and was based on a sample size of 202 patients whose age ranged from 1 to 79 years. It was highest (100%) in the OB/GYN and pediatric clusters and lowest in the surgery (94%) and internal medicine (93%) clusters [Figure 3]. Ultrasound and CT scored best (100%) among imaging techniques and conventional radiography scored least (94%) [Figure 4]. Normal imaging results were utilized in 96% of cases and abnormal in 97%. The imaging results led to change in treatment in 68% of patients, modification of treatment in 25% and requisition of additional imaging modalities in 6% of the patients.

Close

Figure 3

Clinical utilization of imaging results (CUI) within clusters. CUI was highest in the obstetrics and gynecology, followed by surgery. It was lowest in the pediatrics cluster.

Export to PPT

Close

Figure 4

Clinical utilization of imaging results (CUI) and types of imaging. CUI was highest for contrasted studies followed by ultrasound and lowest for CT/MRI.

Export to PPT

The validity of the clinician in identifying correctly those patients who need imaging (sensitivity) and those who do not (specificity)

Correct identification by the clinician, of patients who needed imaging is equated to the sensitivity of a test and was 85%, whereas specificity was 92%. The positive predictive value was 94% and negative predictive value 80%. The positive likelihood ratio was 10.6 and negative likelihood ratio 0.16.

The qualitative component of the study

Ways to improve requisitions

The interviewees proposed ways of improvement are categorized as follows: care while conducting physical examination of the patients, care in writing requisitions, ensuring appropriate choice of the examination, and better support supervision for junior doctors.

Deficiencies and problems with imaging reports

The deficiencies were grouped under categories: varying report styles, unfamiliar terminology, brief descriptive section, discrepancies within imaging reports, inconclusive reports, unfocussed differential diagnosis, and irrelevant recommendations.

Within the category of varying report styles were lack of uniformity and unfamiliar terminology. One said: “There are some terms that you use, unfortunately, we cannot understand them, like echogenicity”. Unfamiliar terminology was with the ultrasound, CT, and magnetic resonance imaging (MRI). This issue raised passionate arguments with some radiologists advocating for use of terminology, especially in the descriptive section of the report. One radiologist said: “I think there is a discrepancy between the expectations of the clinicians and what the radiologist provides in the report. The radiologist cannot make an exhaustive description without using radiology words”.

There was a complaint over brief descriptive section. One criticized: “so, if the report is very short and it just jumps to conclusions, you start asking yourself how one came to those conclusions without describing exactly what they found”.

Discrepancies with clinico-laboratory data were pointed. Some doctors complained that some reports were inconclusive, hence, as one stated: “leaving me in limbo as to what the problem and appropriate intervention could be”.

Views on improving reports related to user competence

Views were advanced by clinicians on how to improve reports. These are categorized as: use of familiar language and correct grammar, clear and uniform reporting style, detailed description, conclusive reports, focused differential diagnosis, care while giving recommendations, and stating limitations of the examinations.

Radiologists and clinicians agreed that some terminologies were unavoidable in the descriptive section but should be avoided in the conclusion. An uniform reporting style was recommended. The descriptive section of the report should be detailed, clear, and tally with the clinical picture.

There were contrasting views on the conclusive part of the report with some in favor of a conclusion and others were not. It was suggested that the differential diagnosis be more focused and preferably include not more than three diagnoses.

The section on recommendations was a subject of much discussion. One clinician raised the eyebrows of the radiologists participating in the FGD by stating: “Sometimes I act on them, sometimes I do not; it depends on what extra information I need, so often I do not always follow them”. Another responded: “if there are recommendations to be made, they should be made as suggestions, but not strong recommendations”. One surgeon ushered in calm by advising that: “Such recommendations are quite useful and we as surgeons respect them and they guide us”.

Appropriateness of imaging decisions

The overall ASU of 94% obtained in this study is higher than that of Kahn et al.,[10] who recorded appropriateness levels of up to 78% among imaging requisitions from a general internal medicine clinic in the USA.

The high ASU is an indication that the clinician fairly accurately predicted those patients who would benefit from imaging and this may be due to pressure from limited resources. Consequently, more expensive investigations like CT, MRI, and fluoroscopy carry a higher ASU. A relatively low ASU for the surgical cluster may be attributable to inappropriate requisitions for trauma cases. This calls for appropriateness criteria to improve on the ASU.

The higher ASU for the more senior clinician illustrates the importance of clinical aptitude and experience in making appropriate requisition decisions though this is not always the case.[11]

Requisitioning and report-writing guidelines may improve performance but their effect is not uniform.[12]

Appropriateness of non-imaging decisions

The overall ANID for Mulago, of 80%, may be explained by the caution the clinician exercises while requisitioning for imaging in the light of limited imaging resources. The lower ANID for surgery may be because many imaging studies are ordered in the emergency room and this is usually by junior doctors. Assessment of the ANID is important in making sure that all patients who would benefit from imaging receive the service and in monitoring inappropriate requisitions.[13]

The clinical utilization of imaging

A CUI of 97% may reflect the performance of the referring clinician and the radiologist. It is interesting to note that both negative and positive imaging results impact patient management. The high CUI is evidence that the radiologists take care to issue an accurate report and that the clinician utilize the results for subsequent patient management.

The CUI may reflect how the subsequent management intervention is solely dependent on the imaging finding hence it is higher for OB/GYN and pediatrics. It is interesting that ultrasound and CT had the highest CUI. It is possible that in these particular clinical scenarios, ultrasound had a high utility.[14]

The validity of the clinician as a measuring tool, in deciding which patients would benefit from imaging

The positive likelihood ratio of 10.6 and negative ratio 0.16 show that the clinician is a good measuring tool for identifying patients who may benefit from imaging in comparison to those who will not.

Problems with imaging requisitioning and proposed solutions

The main problems identified and the causes are similar to those by other researchers.[1516] Solutions like enhancing interactions between radiologists and clinicians plus provision of imaging guidelines have been proposed by others.[17–19]

Deficiencies with imaging report and how to rectify them

The study revealed areas of deficiencies in imaging reports, which included incomprehensive terminologies, nonauthoritative reports, and inconclusive reports. The clinicians suggested improvements. Other authors have advocated for promotion of professionalism and communication skills for radiologists so as to improve patient satisfaction and outcomes.[5]

In this study, ultrasound, CT, and MRI were found to have the most incomprehensive terminologies. Other authors noted variability of readability indices with different procedure.[20]

This study suggests limiting terminology to the descriptive section, and using ordinary language in the conclusion. Use of proper descriptive terms has been recommended by others.[2122] This study, like others reveals that clinician want more descriptive detail.[2123] Clinicians asked for more authoritative reports, an observation also made by other workers.[20] This study findings highlight the importance of the conclusion, and other authors concur with this observation.[21] The importance of limited and pertinent differential diagnoses has been underscored by others.[21] This contention and debate on recommendations is not new in literature.[2324] Recommendations should be to the patient's interest, not overzealous, and with a clinical basis.[17]