Introduction

The practice of radiology has shifted from the interpretation of hard-copy images to soft-copy digital images.1 Picture archiving and communications systems (PACS) permit integration of digital images with the power of computers to bring important benefits to radiology services and have the potential to contribute substantially to their future development.2 This has led to the wide-scale adoption of PACS.3 Currently identified benefits include improved patient care through better organized patient scheduling that reduce wait times, quicker turnaround for radiologists’ reports, better communication with referring physicians, accurate diagnosis with easy onsite and remote access to current and archived images and reports for comparison, access to automated transcription and computer-aided image enhancement/detection, and overall better service to patients and physicians.4,5

Selection of the right PACS for an institution poses many challenges given the numerous stakeholders, vendor options, and criteria that need to be met. Among them, radiologists are the most important users and stakeholders. Despite the importance of the selection of the right PACS, there have not been concerted efforts to empirically assess the relative importance of various criteria in selecting a PACS in prior research, though various criteria have been suggested.1 This study investigates radiologists’ preferences and requirements for a good system and employs analytical hierarchy process (AHP) to develop priority weights and rank them based on a survey of radiologists. Analytical hierarchy process was developed by Thomas Saaty6 to guide multiple-criteria decision making; it is one of the most widely used methods of analysis7 for decision making today.

Methods

Analytic Hierarchy Process

AHP method was employed to examine the relative importance of PACS’ functional and technical selection criteria or dimensions. AHP requires the decomposition of a goal into a homogenous set of dimensions (sub-goals or criteria).8 Each dimension can be further divided into sub-dimensions to develop an analytical hierarchy. Respondents are asked to make pairwise comparisons of dimensions at the same level and rate their relative importance, in each hierarchical path, to determine the relative priority weights for each dimension.9 Thomas Saaty6 developed consistency ratio (CR) to evaluate the consistency of responses (or transitivity) within a respondent and across respondents, where CR value of 0.1 or less indicates good consistency (90% confidence) and values beyond 0.1 represent weak consistency. CR is not applicable if only two dimensions are being compared as transitivity would not be an issue.

Development of the AHP Model

In this study, the goal of the AHP was PACS selection. The analytical hierarchy incorporating the identified dimensions is presented in Figure 1. To identify the dimensions for PACS selection, a literature review was conducted, and the identified features/criteria were organized into main dimensions at level 1 and sub-dimensions at level 2 and level 3 as per the AHP method. Thereafter, three radiologists and three PACS administrators were consulted to seek feedback on the identified dimensions and refine the hierarchical organization of the dimensions through an iterative process. The AHP dimensions identified from literature review and feedback from radiologists and PACS administrators are explained below.

Fig. 1
figure 1

Analytical hierarchy for PACS selection criteria dimensions.

Full size image

Display quality refers to the ability of PACS to store and render images of the required quality (resolution and pixel depth) for different modalities consistently over time. Display quality was considered as a level 1 dimension since it is critical for the accurate interpretation of radiographic images and diagnosis.10 For display quality, three level 2 sub-dimensions were identified: support for high image quality,11 built-in tools for quality assurance,10,12 and support for multi-modality images on the same workstation.11

User interface for image manipulation refers to the PACS user interface that permits radiologists to conveniently manipulate and view images. Since the design of the interface may impact their productivity and work-related fatigue/stress, user interface for image manipulation5,13 was identified as a level 1 dimension. Four level 2 sub-dimensions were identified that can contribute to a better user interface: easy-to-use hanging protocol and icons,14 customizable hanging protocol and icons,13 convenience and responsiveness in image manipulation,11,13 and computer-aided diagnosis (CAD) and visualization.5,15,16 For the customizable hanging protocol and icons sub-dimension, two level 3 sub-dimensions were identified: institutionally customizable and individually customizable hanging protocol and icons.14 In the case of “CAD and visualization” sub-dimension, two level 3 sub-dimensions were stand-alone facility and integrated with workstation.17,18

The third level 1 dimension was user interface for workflow management.11,13,19 It refers to the tools provided by PACS for organizing the radiology work by body area and modality, improving the throughput, and reducing average and emergency turnaround times. To achieve these goals, initially, two level 2 sub-dimensions, worklist and workflow organization17,19 and voice recognition,5 were identified. In addition, case schedule and backlog monitoring were added based on the consultation from radiologists and PACS administrators. Worklist and workflow organization can be integrated either with PACS or radiology information systems (RIS),19,20 which constituted the two level 3 sub-dimensions. Similarly, voice recognition sub-dimension was also further divided into three level 3 sub-dimensions based on literature review: auto-transcription,21,22 third-party transcription,23 and structured reporting.19,24 However, it should be noted that third-party transcription was dropped from the main analysis, though preferences for this criteria were gathered, as it was not considered strictly a PACS feature based on PACS administrators’ feedback.

The fourth main dimension identified was system performance and architecture.12 It refers to PACS’ performance in the retrieval of short-term and long-term archived images and the flexibility for adding new features and products. For this dimension, three level 2 sub-dimensions were identified, which can help improve overall PACS performance: compatibility and integration with other systems and products,12,19,20 retrieval speed for short-term and long-term archived images,12 and tools for continuous PACS performance monitoring.12 The first sub-dimension dealing with compatibility and integration was further subdivided into two level 3 sub-dimensions, open standards and proprietary standards,12,25 which can influence integration of third-party products and reliance on a single vendor.

The fifth and final dimension was system continuity and functionality.12,17,22 This dimension reflects the institutional requirements for continuity and security as well as functionality for multi-specialty images. Three level 2 sub-dimensions were identified for this dimension: support for service and configuration/upgrades12; security, backup, and downtime prevention12,26; and support for multi-specialty images from outside radiology.27

Survey and Data Analysis

To assess the relative importance (or weights) of PACS selection dimensions identified in Figure 1, a survey was designed to obtain data from radiologists for the AHP method. An online questionnaire was developed based on the identified dimensions (a sample of items used is included in Appendix A), which asked the respondents to rate the relative importance of each dimension with other dimensions at the same level within each branch of the hierarchy. The study protocol and survey instrument were reviewed and found to be exempt from human subject regulations (45 CFR 46.101(B)) under the Human Research Protection Office guidelines of the institutional review board at the Mallinckrodt Institute of Radiology, Washington University prior to data collection.

AHP method was employed to calculate the priority weights for different dimensions within each level of the hierarchy. The global priority of each sub-dimension was calculated by taking a product of the priorities of higher-level dimensions in the path as depicted in Figure 1. The priority weights reflect the overall preference or importance assigned to a dimension relative to the other dimensions. AHP can be employed on a small set of responses, with sample size as low as 20.7–9

Results

Sample

Links to the online survey were posted in general radiology discussion forum at AuntMinnie as well as e-mailed to randomly chosen 490 radiologists from the RSNA directory in North America. Out of the e-mailed questionnaire, only 435 were successfully delivered due to errors in e-mail addresses or difficulties in getting through host mail servers. Five responses were received from the discussion forum. An additional 53 responses were received after e-mailing the survey participation request, yielding a total of 58 responses and a survey response rate of approximately 12% (53 out of 435 delivered). After setting aside 11 responses for incomplete data, 47 responses were used for further analysis. The number of respondents obtained in this study is adequate compared to other AHP studies, as AHP does not require large sample sizes.28

Analysis of demographic data revealed that the respondents had an average 12.30 years of experience in radiology, 7.96 years of experience in using PACS, and 20.40 years of experience in using computers. On a four-point scale for the level of involvement with PACS selection/administration (1, little/none, to 4, very active), the average level of involvement was rated at 1.77. The average frequency of different types of images seen was as follows on a five-point scale (1, rarely, to 5, most of the time): cross-sectional (4.47), plain film (3.78), fluoroscopy (2.70), angiography (2.65), nuclear, PET (2.48), and mammography (2.32). Out of 47 respondents, six had engineering or computer-related degrees and the rest in science or arts. Thus, it appears that the survey included a range of experienced radiologists with reasonable experience in the use of PACS and involvement in its selection/administration.

AHP Results

The first step in the AHP analysis was to assess the relative priorities of the main dimensions at level 1 in the analytical hierarchy. The results are presented in Table 1. Next, the relative priorities of level 2 and level 3 sub-dimensions within each higher dimension were analyzed. These results are included in Table 2. Good consistency ratios (below 0.10 level as recommended) were obtained for all the included comparisons as reported in Tables 1 and 2. This suggests that the opinions expressed by radiologists are not arbitrary.

Table 1 Priority of dimensions at level 1 of AHP
Full size table
Table 2 Priority of sub-dimensions at level 2 and level 3
Full size table

In order to be able to compare the relative importance of different sub-dimensions across dimensions, we computed the global priority of each sub-dimension by taking a product of the priority of the sub-dimension and its higher-level dimensions in the hierarchical path. The global priorities for the level 2 and level 3 sub-dimensions are presented in Table 3 after rescaling the fractional weights by multiplying with 100.

Table 3 Global priority weights for level 2 and level 3 sub-dimensions (scaled to 100)
Full size table

It may be noted that the third-party transcription sub-dimension was not included in the AHP analysis as it is not a PACS feature. However, we had asked radiologists to indicate preference for automated transcription vs. third-party transcription. The results indicate that radiologists strongly preferred third-party (manual) transcription over automated transcription, with priority weights of 0.67 and 0.33, respectively.

Discussion

The results suggest the importance of issues in PACS development and selection from the perspective of radiologists, who are the principal users of PACS. The priority weights reported in the paper may be useful in the PACS selection process for ranking different PACS products by taking a weighted average of their assessed scores on different dimensions. Radiologists assigned the highest importance to system continuity and functionality in selecting a PACS with a weight of 0.32 as reported in Table 1. The high importance assigned to system continuity and functionality reflects the criticality of radiology services for patient care. The next dimension in importance was systems performance and architecture with a weight of 0.23. This can ensure faster access to short-term and long-term stored images. The first two dimensions are critical for making productive use of radiologists’ time and for maintaining high throughput to complete the pending workload in normal working period.

The third important dimension was user interface for managing workflow with a weight of 0.20. It helps radiologists to better organize their work schedule and permits them to focus on cases from their area of specialization as well as those needing urgent attention. This dimension also helps achieve better-managed departmental workflow and caseload. The fourth important dimension was user interface for image manipulation with a weight of 0.17. Good user interface for image manipulation and viewing is important for radiologists in conveniently viewing images from different modalities. It can reduce fatigue and stress related to long hours spent in using pointing devices for examining images. Finally, display quality received the lowest priority from radiologists with a weight of 0.08. It could be a reflection of the fact that most recent PACS products provide good support for storing and displaying high-quality images. Thus, overall radiologists gave higher priority to the need for business continuity and functionality, followed by requirements in terms of performance, speed, and throughput, and gave them priority over their interface preferences/requirements for convenience and ease of use.

When the priority weights for sub-dimensions are considered, next to the mandatory requirements of security, backup, and downtime prevention, radiologists assigned high priority to the emerging developments in radiology. For example, voice recognition, transcription, and reporting were given very high priority. These features belong to user interface for workflow management and can help improve the overall institutional throughput and customer service with clear communication of their interpretation and faster turnaround times. In this area, structured reporting was also given a relatively high priority.

Tools for continuous PACS performance monitoring, support for service and upgrade, and retrieval speed for short-term and long-term archived images were also rated very high. These dimensions can help ensure high performance and throughput and provide warning of possible impending breakdowns to help rapidly fix them to ensure continuity, which is critical for maintaining high productivity and completing the pending work during normal working hours.

In terms of user interface for manipulating images, radiologists gave the highest priority to convenience and responsiveness in the manipulation of images. Easy-to-use hanging protocol and icons and individually customizable interface were also emphasized. One comment was the need to avoid long movements of the cursor (from one monitor to another or within a screen) and precise adjustments to the cursor position, which can be stressful. Further research attention is needed for the development of better user interface and pointing devices, especially for the unique multi-monitor usage pattern associated with PACS. For example, a simple toggle (e.g., function key) may permit users to jump from one monitor to a preferred location on the other monitor without long hand movements. The need for very precise mouse movements may be avoided by programming the arrow keys to permit precise adjustments in conjunction with some function key once the cursor is locked on the object to be manipulated. Moreover, while permitting the viewing of images at the eye level, all the image manipulation controls may be grouped within the bottom (or top) two or three inches of the screen in an image manipulation interface ribbon, including the ability to zoom, shift, and move across cross-sectional images via a small copy of the main image. This may help reduce the fatigue and stress associated with hand movements during image manipulation. Radiologists also preferred support for multi-modality images and CAD and 3D visualization to be integrated on their workstation, instead of separate stand-alone facilities. Some of the above features and the high priority assigned to responsiveness to image manipulation suggest the need for fast, high-performing PACS workstations for radiologists.

Radiologists were found to assign much higher priority to the integration of worklist and workflow with PACS, compared to their integration with RIS. In an ideal situation, this should not be an issue if PACS and RIS are tightly integrated. However, in its absence, radiologists’ preference for integration with PACS may reflect the availability of more accurate and up-to-date caseload and imaging procedure status information in PACS.

Strong preference indicated by radiologists for manually performed, third-party transcription over automated transcription (0.67 vs. 0.33) by a margin of 2 to 1, may be indicative of the weaknesses in the automated transcription in many PACS, which can necessitate additional effort and time from radiologists to edit and correct the errors and conforms to the findings of other researchers.29 This weakness is further compounded when the best of breed automated transcription products available from third-party vendors cannot be readily integrated with PACS products due to their proprietary architectures. Thus, structured reporting and automated transcription are clearly areas of interest to radiologists in which PACS vendors have to make significant improvements.

With regard to compatibility and integration with other systems and products, which is a sub-dimension of system performance and architecture, a majority of radiologists favored open standards over proprietary standards. Open standards in PACS will permit adoption and convenient, more robust integration of the best of the breed third-party products in areas of automated transcription, 3D visualization, image enhancements, computer-aided detection, and other advances such as structured reporting. Open standards will also offer flexibility to institutions to change their PACS vendors without incurring overbearing conversion costs. To some extent, the results may also be a reflection of the institutional affiliation of the respondents based on the expectation that open-system architectures can provide greater benefits to large institutions, which can afford to keep their own skilled information technology (IT) and PACS administration staff to take advantage of their flexibility, and be able to integrate the products from different vendors. On the other hand, smaller institutions with fewer IT and PACS professionals may prefer to rely on the PACS vendor to provide an integrated solution, without creating the need for high-level in-house expertise and effort.

Conclusions

Overall, this study provides useful insights into radiologists’ preferences for PACS features. The results should be useful in PACS selection and architecture design as well as for vendors in indentifying radiologists’ needs and preferences in developing better products. Good consistency ratios give confidence in the results that the responses are not arbitrary. Results also show that radiologists’ are now less concerned with general image quality and manipulation and more concerned with broader institutional and workflow issues. Radiologists gave the highest priorities to the needs of patients and clinicians in terms of security and prevention of any loss of data, fast turnaround times (via automated transcription and fast access to images), better reporting format and communication (structured reporting), and the need for business continuity and high throughput. Emerging advances in radiology such as automated voice transcription and structured reporting were considered important discriminators by radiologists in PACS selection, next to the mandatory requirements such as security, backup, and business continuity.