Introduction

Picture archiving and communication systems (PACS) have emerged as one of the most important innovations in the practice of radiology. These systems leverage the extant computer and communication technologies to capture, store, and deliver patient imaging studies to radiologists [1]. Some of the benefits of PACS systems that have been identified and discussed in the literature include convenience in storage and organization of an ever-increasing volume of imaging studies, easier access to a vast pool of the current and prior imaging studies, faster and more efficient workflow organization, effective management of skills and resources, and convenient access to computer-aided visualization and diagnostic tools [2, 3]. Successful implementation of PACS is critical to realize the wide range of benefits they can provide in terms of productivity, quality of diagnosis, turnaround time, and the quality of service to patients and referring physicians [1–4].

PACS technology and products are being constantly upgraded and refined. While some organizations may be selecting their first PACS installations, others may be upgrading their current PACS or investing in a new system. A large number of vendor offerings and architectural choices make the task of selecting a PACS system difficult. The selection process may be further complicated by the different interests of stakeholders such as PACS administrators and radiologists, and others (e.g., hospital administrators, clinicians, and imaging technicians). While radiologists may be focused on convenience in usage and workflow, PACS administrators may be more interested in performance, monitoring tools to ensure continuity, vendor support, and integration and compatibility with the currently deployed technologies. In particular, some of the integration, performance, and support issues, if not addressed at the selection stage, may pose significant implementation and operational challenges for the administrators, who are likely to be responsible and held accountable for delivering an integrated, high-performance system in a timely manner. Thus, the selection process can pose a challenge for both groups in arriving at a choice that can meet all the requirements. Some of these difficulties may be mitigated by better understanding the perspectives of the two groups. As radiologists and PACS administrators have to collaborate to arrive at a good choice in the selection and implementation of a PACS, clarifying and understanding the expectations and requirements of the two groups can be useful in the decision-making process. Differences in perceptions and a lack of understanding of the requirements of the other side may cause friction and suboptimal decisions. Additionally, PACS administrators may also have to meet the requirements passed down by the hospital administration in terms of requirements for integration with other systems and support for non-radiology specialty images. Often, PACS administrators may have to coordinate and harmonize the interests of different stakeholders to arrive at a reasonable decision. To aid in better understanding the requirements from PACS administrators’ perspective, this paper presents a study that identifies and ranks the features sought in PACS systems by PACS administrators. It develops an analytical hierarchy [5] based on the data obtained through a survey of PACS administrators. The weights assigned to different features by PACS administrators are computed and compared with the previously reported data obtained from radiologists. The paper analyzes the results and presents a discussion of the important PACS features identified in the study. It also discusses the similarities and differences in the perceptions of the two groups regarding the importance of PACS features before concluding with some recommendations. The analysis and priority weights presented here may be useful in the development and selection of PACS.

Methods

Analytical Hierarchy Process

The analytical hierarchy process (AHP) is one of the most widely used analysis techniques for multicriteria decision making [5–8]. It provides a means to classify various criteria that are relevant in arriving at a decision and to develop importance weights for them based on a survey of the relative preferences of the target population. The criteria for the decision goal are classified into main dimensions and subdimensions in the form of an analytical hierarchy. A survey of the domain users (or experts) is conducted to obtain pairwise comparisons of dimensions at each level within each hierarchy path. These comparison rankings are aggregated and analyzed to arrive at the relative weights for each dimension in the hierarchy as per the AHP. The validity of the results is assessed based on the consistency of responses, which is assessed through a consistency ratio (CR) developed by Saaty [5], where a CR value of 0.1 or less indicates good consistency (90 % confidence) and values above 0.1 indicate weaker consistency.

AHP Model for PACS Administrators

A review of the PACS literature and interviews with three radiologists and three PACS administrators yielded a set of relevant criteria that were considered important in PACS selection. The relevant criteria for radiologists and their importance have been noted in the literature [9]. In this paper, we extend the AHP to include the criteria of interest to PACS administrators and report on the results of a survey to assess the importance weights assigned by them to the multiple criteria through an AHP.

Based on the additional unique inputs from PACS administrators, an analytical hierarchy was developed for the PACS selection criteria consisting of main dimensions and their subdimensions as shown in Fig. 1. It consists of five main dimensions: display quality, user interface for image manipulation, user interface for workflow management, system performance and architecture, and system continuity and functionality. Display quality is important for accurate diagnosis and reading of the images. Display quality was identified to have three subdimensions at level 2: support for high image quality [10], built-in tools for quality assurance [11], and support for multimodality images on the same workstation [12].

Fig. 1
figure 1

Analytical hierarchy for PACS selection

Full size image

The design of a good user interface with ease of use is important for radiologist productivity, accuracy, and fatigue [3, 13]. The user interface for image manipulation was identified to have four subdimensions at level 2: easy-to-use hanging protocol and icons [14], customizable hanging protocol and icons [13], convenience and responsiveness in the manipulation of images, and computer-aided diagnosis (CAD) and visualization (e.g., 3D) [3, 15, 16]. Among level 2 subdimensions, the customizable hanging protocols dimension was divided into two subdimensions at level 3: institutionally customizable and individually customizable [14]. Similarly, computer-aided diagnosis and visualization was further divided into stand-alone facility and integrated with work station [17].

Workflow management is important for organizing radiology work by modality and improving turnaround time and throughput [12, 13, 18]. The user interface for workflow management was identified to have three subdimensions: worklist and workflow organization [17, 18], case schedule and backlog monitoring, and voice recognition, transcription, and reporting. Worklist and workflow organization consisted of two level 3 dimensions: integrated with radiology information system (RIS)/RIS and integrated with PACS/PACS [18]. Voice recognition, transcription, and reporting [3] was divided into two level 3 dimensions: auto-transcription [19, 20] and self-editing and structured reporting [18, 21].

High performance and a good architecture are important for rapidly expanding radiology image volume and corresponding storage organization and fast retrieval speeds [22]. System performance and architecture was identified to have three subdimensions at level 2: compatibility and integration with other systems and products [18, 22], retrieval speed for short- and long-term archived images [22], and tools for continuous PACS performance monitoring [22]. Compatibility and integration with other systems and products was divided into two subdimensions at level 3: open standards and proprietary standards [22, 23]. Especially for PACS administrators, two additional subdimensions were identified at level 3 for tools for continuous PACS performance monitoring: real-time monitoring and periodic/trend reporting.

Business continuity and security are vital for delivering high-quality service and patient care [17, 22, 24]. A loss of operations can severely disrupt the entire radiology cycle. Additionally, supporting non-radiology images is also becoming important in modern health-care systems. Thus, system continuity and functionality was identified as an important dimension. It consisted of three subdimensions at level 3: support for service and configuration/upgrades [22]; security, backup, and downtime prevention [22, 25]; and support for multispecialty images (non-radiology images) [26, 27]. For PACS administrators, the last dimension was divided into cardiology images and pathology images at level 3. In the interviews, it was stated that outside of radiology, the most active consideration for storing other specialty images was for cardiology and pathology. Therefore, these two were included in the hierarchy.

Survey and Data Analysis

A survey instrument was developed to obtain the pairwise preference responses of PACS administrators for the dimensions at each level in the hierarchical paths of the analytical hierarchy. Survey participants were assured of complete anonymity. The study protocols, instructions, and instruments were reviewed by the institutional review board office at Mallinckrodt Institute of Radiology, Washington University, and found to be exempt from human subject regulations as per Human Research Protection Office guidelines (45 CFR 46.101(B)) [28]. An online survey link was emailed to 422 PACS administrators with an invitation to participate in the research. The respondents were identified through PACS user/discussion groups and RSNA directory. The survey yielded 68 responses with a response rate of 16.1 %. Out of 68 responses, after eliminating incomplete and few inconsistent responses, 48 responses were used for the AHP to compute the priority weights (representing importance) for different dimensions at levels 1, 2, and 3. The response rate of about 16 % for the PACS administrators’ survey was slightly better compared to the response rate for radiologists at 12 %. The response rates below 20 % for web-based surveys of health professionals are not uncommon [29]. The sample size is adequate for AHP analysis, which does not require large samples [30].

Results

The data obtained from the sample was analyzed using AHP to determine the relative weights (or importance) of different dimensions (or criteria) in the analytical hierarchy. The results for the first level in the hierarchical path are presented in Table 1 in the Appendix. The results showed a good consistency with a consistency ratio of 0.086, which provides more than 90 % confidence level. The results for the second and third levels in the hierarchy are presented in Table 2 in the Appendix. The consistency ratios for different paths range from 0.005 to 0.092, suggesting well over 90 % confidence levels. Good consistency ratios suggest that the responses of the PACS administrators are not random but may be well thought out responses. Thus, it gives confidence in the results obtained in the AHP analysis. To permit comparison of individual dimension across different hierarchical paths, global priority weights were also computed for each dimension by taking a product of the weights assigned along the hierarchical path leading to it. The overall weights for each individual criterion are presented in Table 3. For comparison with the previously reported results for radiologists [9], the corresponding results for administrators and radiologists are placed together in Tables 1 and 2, and the overall weights for radiologists are reproduced in Table 4.

Sample

The sample represented a wide range of imaging professionals from varied backgrounds. The average experience of respondents with PACS was 8.37 years, which shows that they were well experienced and knowledgeable about the issues raised in the survey. These respondents were also highly involved in PACS selection as shown by an average response of 3.47 for the level of involvement on a scale of 1 to 4. Nearly 100 % of respondents were involved in PACS administration, rather than radiology business management. In terms of educational backgrounds, undergraduate degree holders included 27 from imaging technology, 5 from engineering, 19 from computer/IT, 15 from science, 1 from arts, and 2 from other backgrounds. In terms of graduate degrees, eight were from computer/IT, seven from health-care administration, four from medicine, four from arts and sciences, and eight from other backgrounds. Thus, the respondents appear to have a high level of involvement in PACS selection and an adequate level of experience in PACS administration, given the age of the technology.

AHP Results

As noted earlier, an analytical hierarchy was developed for the relevant (criteria) dimensions for PACS selection that were identified through literature review and interviews with three PACS administrators and three radiologists. Survey responses from 48 PACS administrators were utilized for the AHP analysis.

System continuity and functionality was found to be the most important dimension, followed by system performance and architecture, user interface for workflow management, user interface for image manipulation, and display quality, with priority weights of 0.37, 0.30, 0.16, 0.11, and 0.06, respectively (see Table 1). Among the subdimensions, the top features were security, backup, and downtime prevention (0.174); tools for continuous PACS performance monitoring (0.141); support for multispecialty images (0.118); and voice recognition, transcription, and reporting (0.102). Support for cardiology images (0.086), retrieval speed for short- and long-term archived images (0.081), tools for real-time monitoring (0.080), and compatibility and integration with other systems (0.078) were also assigned high-priority weights (see Table 3).

In some of the areas of primary interest to administrators, the results indicate their preferences (see Table 2). In systems performance and architecture, administrators gave the highest priority to tools for continuous PACS monitoring (0.47), followed by retrieval speeds for archived images (0.27) and compatibility and integration with systems (0.26). Under compatibility and integration, there was overwhelming support for open systems over propriety (vendor) standards (0.69 vs. 0.31). Administrators preferred real-time monitoring tools over periodic/trend reporting on the status of PACS (0.57 vs. 0.43). In terms of support from vendors, administrators assigned the highest importance to support for system upgrades (0.54), followed by support for ongoing configuration (0.28) and service events (0.18). Administrators also assigned greater importance to the storage of cardiology images over pathology images by 2.7 to 1 (0.73 vs 0.27), which may reflect demands from non-radiology specialties and institutional plans and priorities communicated to them.

In systems continuity and functionality, administrators assigned the highest priority to security, backup, and downtime prevention dimension (0.47), followed by support for multispecialty images on PACS (0.32) and support for service and upgrades (0.20). Administrators gave higher priority to the storage of cardiology images over pathology images by a factor of 2.7 (0.73 vs. 0.27). In support for service and upgrades by the vendor, the highest priority was assigned to help with system upgrades (0.54), followed by ongoing configuration (0.28) and service events (0.18).

Outside of the AHP dimensions, we had also asked respondents to compare third party transcription services with automated voice transcription, and third party transcription was favored 2 to 1 (0.67 vs. 0.33). Thus, it seems that automatic transcription systems need improvement. Users may also not be able to get the best of breed transcription software integrated into their PACS given the preponderance of propriety PACS architectures in the market. Thus, the high priority assigned to open standards over proprietary (0.69 vs. 0.31) appears to be reasonable.

It is reassuring to note in Table 1 the overall correspondence in the relative ranking of the five main dimensions at level 1 by PACS administrators and the results for radiologists in previous research. Both groups assigned the highest importance to system continuity and functionality, followed by system performance and architecture. Display quality was assigned the lowest importance by both groups.

As noted in Table 2, among level 2 and level 3 subdimensions, as compared to radiologists, PACS administrators gave somewhat higher importance to built-in tools for image quality assurance over support for high image quality. However, both groups gave the highest importance to the requirement of the ability to view multimodality images on the same workstation. There were only minor differences in the importance assigned to CAD and visualization, where radiologists gave a greater importance to this feature compared to PACS administrators. PACS administrators gave somewhat greater importance to support for multispecialty images compared to radiologists.

Overall, there is a high degree of correspondence between PACS administrators’ priorities with that of radiologists’ priorities. PACS administrators gave the highest priority to security, backup, and downtime prevention; tools for PACS performance monitoring; support for multispecialty images; and voice recognition, transcription, and reporting dimensions. Radiologists’ top four ranked dimensions included three of PACS administrators’ top preferences and excluded multispecialty images, which were replaced by structured reporting (see Table 4).

Discussion

The results point to the dimensions that can be critical discriminators between different PACS and highlight the importance of faster integration of the emerging developments in radiology into PACS. It was assuring to note that PACS administrators gave similar ranking to main PACS dimensions as radiologists. The highest priorities were assigned by both groups to business continuity and functionality, followed by systems performance and architecture and workflow management. Both groups assigned lower priorities to display quality and user interface for image manipulation. It appears that these factors are no longer considered to be critical discriminators between different PACS products, given the relative high levels of performance achieved on these dimensions compared to other dimensions.

There were some differences in the priorities of administrators and radiologists in the lower-level dimensions. Both groups assigned the highest priority to security, backup, and downtime prevention and very high priority to tools for continuous PACS administration. Thus, the importance of the institutional need for continuity in service to patients was understood by both groups and was given the highest importance by both of them. PACS administrators’ next high priorities were support for multispecialty images, retrieval speeds, real-time monitoring and architectural issues of compatibility, and integration with other products which focus on their ability to deliver services and support. On the other hand, beyond continuity of services, radiologists gave high priority to voice recognition, transcription, and reporting; structured reporting; and convenience and responsiveness in manipulation of images, which are issues that may impact their productivity and convenience. Thus, on the whole, both groups demonstrated unanimity on the importance of the main dimensions and agreed on the critical importance of business continuity, a key dimension for customer service. Among lower-level, individual dimensions, administrators focused on their ability to deliver services and support for PACS operation, while radiologists focused on issues that may impact their productivity, effort, and accuracy.

Both groups also agreed on the importance of open standards for PACS systems and assigned it two times greater importance over proprietary systems. With the emphasis on greater integration between various components within radiology departments, as well as digitization and integration across different functional areas in health care [31], the importance of open standards is rising. Radiology already has a lead in digitization and integration of PACS and RIS, which can be extended to include enterprise electronic medical record systems. The importance of open systems is likely to rise with greater emphasis on seamless integration in health-care services. Open systems may also be more convenient for integration across PACS of different institutions and with external providers of vendor-neutral storage and cloud services. Despite their advantages, the lack of near 100 % support for open systems found in this study may be due to the differences in the preferences of small/medium institutions and large institutions. Small/medium institutions may prefer a single vendor solution in view of their limited technological capabilities, as the responsibility for integration is primarily taken over by the vendor in proprietary systems. However, the institution is locked in with a single vendor, who may have a limited application portfolio and may not provide the best of the breed solution for each application. With open systems, institutions may need greater technological competence to be able to evaluate different vendor options and integrate them with some in-house customization or programming, or through outside consultants. Large institutions can easily acquire the in-house technological competence to adopt open source and benefit from it to meet their rapidly increasing demands for integration across the health-care delivery chain. Thus, it may be useful to investigate the effect of institutional size on preference for open vs. proprietary systems in future research.

There was unanimity among administrators and radiologists about the preference for third party transcription over automated transcription supported by PACS by 2 to 1. Thus, there is a need for the development of better transcription software. More importantly, given that all PACS vendors may not excel in this area, they should adopt open standards or at least provide and support open interfaces for ready integration of the best of breed voice transcription software from vendors focused on this segment.

This study focused on the PACS features sought by two main stakeholders: PACS administrators and radiologists. However, with the increasing role of radiology in patient care, the interests and views of other stakeholders should also be considered in future studies related to PACS selection to obtain a broader view of the criteria for PACS selection and acquisition. For example, hospital administrators are interested in efficient data and quality metrics reporting from radiology for ensuring safety and effective patient care [4]. Similarly, radiology technicians, administrative staff, clinicians, and others may also have interests that should be considered in PACS selection. Future research should also consider additional features that may gain relevance with the emergence of new technologies and practices, such as cloud computing and vendor-neutral archives (VNA). There is also likely to be a need for some level of interfacing or integration with other institutions’ PACS.

Conclusion

This study identified a set of criteria for PACS development and selection based on input from PACS administrators. A multilevel hierarchy of the criteria was prepared based on literature review and inputs from PACS administrators, and a survey was conducted to obtain responses from PACS administrators for the analytical hierarchy process to identify the relative importance (priority weights) of different dimensions in the hierarchy. Given the multiple, competing requirements for the selection of a PACS, and a multitude of vendors, the priority weights can be useful for developers as well as buyers in decision-making analysis and ranking of products. It was also noteworthy that overall there was a high level of agreement among administrators and radiologists on the overall importance of factors related to customer service: business continuity and systems performance, with few differences in the importance assigned to some of the lower-level dimensions based on their specific concerns: services and support for administrators vs. productivity, effort, and accuracy for radiologists. The research should help promote better understanding of the requirements of the two groups and help identify the relative importance of different features in the design and selection of PACS.