Introduction

Over the last few years, synergies have grown between the information technologies that digitize clinical data and provide access to telecommunication networks and the health care reforms that have tried to improve service access and continuity. This has spurred wider dissemination of computerized systems or picture archiving and communication system (PACS) in hospital medical imaging departments. Several studies have demonstrated that adoptions of this technology have become relatively smooth undertakings, despite the profound changes required to make the transition to digital data.1 The success of these implementations is due to the technical as well as organizational maturity attained by the technology. In terms of the technology, the hardware and software have proven value and have become very reliable. In terms of their place in the organization, system functionalities have achieved a good fit with the clinical business process requirements of medical imaging in hospitals. This guarantees that users will enjoy a range of benefits, something that has been confirmed in many recent studies. A positive impact is reported in terms of better X-ray image quality.2–6 PACS can thus improve the accuracy of clinical diagnoses, so it has a direct impact on quality of care.7 In more general terms, studies show how the new technology transforms work organization and clinical business processes, generating greater overall productivity.8–10 Higher productivity is observed among radiologists11–13 as well as radiological technologists.13,14 These productivity improvements stem from improved work processes,8,15 better data integration,2 a faster production of diagnosis reports,8,15,16 and shorter waiting times and hospital stays.3,15 Finally, economic studies have also been positive, since cost savings have been achieved.15,17–19

With this wide range of benefits, it is easy to understand why many stakeholders suggest wider use of this technology. One promising model is inter-hospital PACS, whereby an interoperable and integrated PACS serves several different hospital sites. Despite the distances between sites, system interoperability allows designers to create a multi-site network for exchanging clinical data in real time. Patient images can be dematerialized and viewed by radiologists in distant health facilities.20 The main impact of this organizational model has been the removal of the barriers in space and time that have traditionally structured professional work in medical imaging. In addition, since the system can integrate several sites, it can radically transform and reorganize how clinical data is managed, determining whether radiologists and technologists are needed on site and how patients gain access to continuity of care. Radiologists and technologists can intervene from a distant location, such that a remote radiologist can make an intervention with a patient who is then served in their local facility. This virtual and broader organization of medical imaging presents new opportunities for managing professional staffing shortages and reduced physical access to medical imaging services. This model is different from teleradiology, since it goes beyond occasional teleconsultations. A virtual medical imaging department is an innovative and more demanding organizational model, to the extent that the underlying goal is to achieve a continuous and more advanced organizational integration of human and physical resources, clinical data, and clienteles. Such a department should therefore provide greater benefits.

Yet this virtual model of medical organization is more complex and presents new and major challenges that may represent hurdles to the expected benefits. First, establishing a virtual organization is technically challenging, since it requires a telecommunications network that can continuously operate reliably and well across multiple and distant sites. In terms of the organization of clinical work, business processes and clinical practices need to be standardized across these sites if the clinical data and patients are to be exchanged efficiently. In this regard, a study was made of a virtual department of medical imaging in Australia that uses an integrated PACS/RIS Web application. The results clearly show the impact of these factors on user satisfaction.21

To better understand the kind of benefits afforded by this innovative model for the virtual organization of medical imaging, we conducted a survey of the users of an inter-hospital PACS system working in five health care institutions at eight different sites. These facilities served a population of 524,000 persons residing on a 21,554 km2 territory. At the time of the survey (2007), this well-established system had been up and running for 3 years. The survey sought to understand the perceptions of three groups of users—radiologists, radiological technologists, and medical specialists—of system benefits. In order to perform a thorough analysis of user perceptions, we used a recognized and tested analytical framework to examine a wide range of benefits associated with PACS.22,23 In addition, in order to analyze the specific nature of an inter-hospital PACS application, two types of uses were identified: intra-hospital use (meaning the traditional internal use of a system) and inter-hospital use (meaning use from at a distant site). The following section presents the conceptual model used to develop the survey questionnaire.

Conceptual Model

Since it was first developed in 1992, Delone and McLean’s model22,23 has been used in a wide variety of studies of information systems, including clinical data systems24 and, more specifically, PACS.21,25 This model has a wide variety of dimensions that can be used to make a methodical and thorough evaluation of all the benefits of an information system as perceived by users. The first set of benefits corresponds to the system’s intrinsic qualities: system quality, the quality of the data it generates, and the quality of available technical support. A second set of benefits concerns satisfaction with the system itself and with system use. This set is used to make assessments of general user satisfaction. A third set of benefits consists of those perceived by users at the individual and organizational levels. Figure 1 provides a list of the benefits assessed by this study as well as the assumed relationships between these variables. In the next section, we discuss how these benefits were measured and how their interrelationhips were tested.

Fig. 1
figure 1

Theoretical Research Model Adapted from DeLone & McLean (2003)23 and Paré et al. (2005)25.

Full size image

Methods

Study Sites and Sample

A questionnaire survey was conducted in the first inter-hospital medical imaging network deployed in Quebec’s health care system. With a population of 7.5 million people, the province of Quebec has the second largest health care system in Canada. The five institutions using this inter-hospital PACS network operate at eight sites, have over 8,000 employees and count 208 medical specialists. Two of the institutions are medium-sized, specialized general hospitals located in urban settings. They have 433 and 215 beds, respectively. The other three institutions operate in remote areas far from urban centres, and each has less than 100 beds. In 2003, these institutions jointly purchased a PACS system and the system is still being operated jointly. The product, SYNAPSETM, is marketed by Fuji. At the time of this study, the Quebec’s health system intranet WAN was an ethernet-type LAN extension. Each hospital had a bandwith connection of 10 Mbs to the regionalized technocenter using a hierachical sytem management architecture for long-term storage. The technocenter had a 100 Mbs bandwidth capacity. Using packet prioritization and compression protocols, the informational infrastructure was designed to minimize the impact of large images traveling over low-bandwidth networks. A definable per modality compression format followed by decompression via the software at the workstation reduces transfer time and delays in an inter-hospital medical imaging network environment. At the time of purchase, the inter-hospital PACS system was seen as a tool that would remedy a serious shortage of radiologists and improve access to medical imaging services in outlying regions. Other objectives included improving the quality of medical imaging services and the continuity of care provided by partner facilities. Finally, it was hoped that the inter-hospital network would generate economies of scale through the acquisition and management of a single integrated system.

We conducted a mail survey of all users of the inter-hospital PACS. The targeted population included all radiologists (n = 15) and radiological technologists (n = 106) as well as all the medical specialists in seven different specializations (general surgery, orthopedics, internal medicine, nuclear medicine, pediatrics, respirology, and urology; n = 71) making intensive use of the medical imaging diagnostic service. In total, 192 persons were asked to participate. The survey was conducted between December 2006 and March 2007, 3 years after the end of the PACS implementation. The opinions received therefore came from seasoned users of a system that had reached cruising speed. The study was approved by the relevant university ethics authorities.

Defining the Perceived Benefits

The survey questionnaire (Appendix 1) was based on the DeLone and McLean model.22,23 First, the intrinsic qualities of PACS were measured in terms of the two contexts of use. Seven variables evaluated intra-hospital use: Ease of use (three items), Screen quality (three items), PACS-RIS integration (three items), Reliability (three items), Accessibility (two items), Perceived usefulness (three items), and Response time (two items). Four other variables measured inter-hospital use: Ease of use (two items), Inter-site integration (two items), Accessibility from remote sites (one item), and Response time (one item). The data quality generated by PACS was evaluated through four variables for intra-hospital use: Completeness (one item), Reliability and validity (two items), Availability (one item), and Safety (one item). The same four variables were also used for inter-hospital use, but we added a fifth variable to measure the Quality of inter-site integration of the data generated by the various centers (two items). Quality of technical support was assessed with one variable (five items) concerning the whole system, regardless of whether use was internal or external. As far as technical support was concerned, it was difficult to establish in which context the support was provided. As suggested by the DeLone and McLean model,22,23 we measured overall satisfaction (three items) without distinguishing type of use (intra-hospital or inter-hospital). Use was measured through a self-assessment made by each user. Intra-hospital use was measured with two variables: Frequency of use (one item) and Intensity of use (two items). Inter-hospital use was assessed with a variable measuring Frequency of use (one item). Benefits were measured in terms of improved Productivity (three items) and Quality of services (four items) for intra-hospital use and in terms of improved Productivity (one item) and Access (four items) for inter-hospital use. Finally, we added two variables to DeLone and McLean’s model22,23 that had been used in another study of the benefits of PACS:25 Confirmation of expectations of users (two items) and their intended future use of PACS (two items). All the answers were measured on 10-point Likert scales that ranged from “Strongly disagree” to “Strongly agree.”

Data Analysis

The internal consistency of each scale was validated using Cronbach’s alpha. Bivariate analyses were performed to confirm statistically significant differences between the three groups of users and the two contexts of use (intra-hospital and inter-hospital). T tests were performed using Mann–Whitney Kruskal–Wallis X 2 tests for user type and Mann–Whitney X 2 tests for type of use. Finally, two analyses of multiple linear regressions tested the relationships hypothesized in our theoretical model. An initial regression model tested the extent to which specific variables could explain overall user satisfaction; these variables measured the intrinsic qualities of PACS, the quality of the data generated, the quality of technical support, confirmation of expectations, and user type. A second regression model tested the degree to which variables for overall satisfaction, benefits, and confirmation of expectations could explain users’ intention to use the system in the future.

Results

Response Rate and Respondent Profile

We received 127 valid questionnaires, for an overall response rate of 66%. The rate varied by type of user: we received valid questionnaires from 80% (12/15) of the radiologists, 72% (76/106) of the radiological technologists, and 55% (39/71) of the medical specialists. Table 1 provides the distributions of these three groups by age, sex, and institution.

Table 1 Socio-Demographic Characteristics of Respondents
Full size table

Psychometric Qualities of the Variables

The internal consistency coefficients (Cronbach’s alpha) of the measuring scales proved highly satisfactory. As shown in Table 2, they vary between 0.73 and 0.97, surpassing the desired minimum threshold of 0.70.26 One measure, Ease of use in an inter-hospital use context, fell below this threshold, with a coefficient of 0.64. We nevertheless maintained this variable for our analyses since it approaches the minimum threshold. Also, according to the literature, ease of use and perceived usefulness are the most significant variables explaining new technology adoption.27,28

Table 2 Descriptive Psychometric and Statistical Qualities of the Scales Used to Measure Assessments of PACS by Type of Use
Full size table

PACS Assessments by Type of Intra-hospital and Inter-hospital Use

Respondents’ assessments of PACS varied by type of use. As shown in Table 2, their overall assessment of the system’s qualities was clearly higher for intra-hospital use (7.9) than for inter-hospital use (5.5; p = 0.0000). The highest scores of system quality were in the variables Perceived usefulness and Ease of use for intra-hospital use (8.2 and 8.5, respectively), while the lowest scores were for Accessibility and Response time in inter-hospital use (4.6 and 4.8, respectively)—even falling below the mid-point of our 10-point scale. The overall measure of data quality was also higher for intra-hospital use (8.7) than inter-hospital use (7.0; p = 0.000). Two variables for inter-hospital use gave the lowest scores: Integration of information (6.2) and Availability of information (6.5). In contrast, all variables measuring data quality in intra-hospital yielded scores above 8/10.

Differences in assessments for intra-hospital and inter-hospital use were also found in system use, both in terms of level of satisfaction and number of users. Scores on both of the measures used to estimate intra-hospital use were very high (Frequency of use, 9.5 and Intensity of use, 8.4), while Frequency of use in inter-hospital applications was lower, at 7.0 (p = .000). On the other hand, while intra-hospital use of PACS was widely reported by the survey’s respondents, only 24% (30/127) indicated that they used the system to call up digital images from a site other than their own. Only six radiologists (50%), 16 medical specialists (42%), and eight medical imaging technologists (11%) used PACS for this purpose. The low percentage of technologists can be explained by the fact that the organization of their work continues to be traditional, meaning that they mainly work in activities that take place in their own hospital.

The differences between these two contexts of use were also apparent in our analysis of perceived benefits. Intra-hospital use received positive assessments in terms of Increased productivity (8.4) and Quality of care (7.8), while inter-hospital use received poorer assessments. In the latter instance, Increased productivity scored a low 4.8 (p = 0.000) out of 10. The score for Access to services was also low (5.7), underscoring PACS’ incapacity to fully support a practice occurring at several sites in a virtual network. This poor assessment of perceived benefits echoes the negative assessments reported above of system quality and the data produced in a context of inter-hospital use. This observation was confirmed in the gap between assessments of Confirmation of expectations in intra-hospital use (8.0) and inter-hospital use (5.7; p = 0.000). Finally, despite the differences in the assessments made in the two contexts, users still reported that they believe in PACS, inasmuch as high scores were recorded for Intended future use in both intra-hospital (8.9) and inter-hospital (8.7) contexts.

PACS Assessments by User Group

Table 3 shows the perceptions of PACS reported by the three user groups in terms of work in an intra-hospital context. Even though statistically significant differences between the three groups were not found in the overall measure of system quality, significant differences were observed in five of the variables of the overall measure (Ease of use, Screen quality, Integration, Perceived usefulness, and Response time). With one exception (Screen quality), technologists reported the highest level of satisfaction, followed by radiologists and then medical specialists. The same order was found in scores for the overall measure of data quality (p = 0.006). Similarly, satisfaction with the Quality of technical support was high among technologists (9.1) but markedly lower for radiologists (7.5) and medical specialists (7.9; p = 0.000). The other dimension of the model that provided statistically significant differences was Use, in terms of both Frequency and Intensity. No differences were found in the model’s other variables. Overall satisfaction levels and assessments of system benefits were therefore similar for all three user groups.

Table 3 Analysis of Differences of Opinion of PACS User Groups in an Intra-hospital Context
Full size table

Table 4 shows user groups’ perceptions in inter-hospital use. In this case, due to the limited number of respondents we were unable to test for statistically significant differences among the groups, but the results are nevertheless revealing. First, the differences of opinion between user groups are more pronounced than in the previous table. Overall, radiologists held the most negative opinions, followed by medical specialists and technologists. The lowest scores on system quality came from the radiologists, particularly with respect to system integration and data integration. Indeed, the integration of system components provided the lowest scores on our 10-point scale: PACS/RIS integration (2.8) and dictation integration (1.7). Scores for Data integration (one of the dimensions of data quality) were very low for the data (4.0) as well as the imaging (3.2). Similarly, perceived benefits in terms of Increased productivity received very poor scores from the radiologists (3.5) but also from the other two groups. The radiologists’ score for Confirmation of expectations is still disappointing (3.7), but the corresponding score from the other two groups was 6.0. In contrast, scores for Intended future use of the system were high for all three groups: radiologists (8.0), medical specialists (8.6), and technologists (9.3).

Table 4 Analysis of Differences of Opinion Between PACS User Groups in an Inter-hospital Context
Full size table

Multivariate Analyses

The interrelationships postulated by DeLone and McLean’s22,23 model were tested with regression analyses. Both regression models performed well, explaining 71% and 48%, respectively, of the variation observed in Overall satisfaction and Intended future use. In the first model, the variables System quality, Data quality, and Confirmation of expectations explained much of the variance in Overall satisfaction. The Medical specialists variable was also statistically significant, indicating differences of opinion between this user group and the other two groups. The second regression model revealed that Intended future use of the system can be explained by two variables: Overall satisfaction and Perceived benefits (Table 5).

Table 5 Linear Regressions of Overall Satisfaction and Intended Future Use of PACS
Full size table

Discussion

The goal of this study was to analyze the range of benefits provided through the virtual organization of medical imaging, which brings together resources from several health care institutions in a single PACS system. This new organizational form is achieved by deploying an interoperable and integrated PACS that allows several remote sites to function as a single entity, sharing equipment, clinical teams, and data and providing care to a common clientele. We conducted a survey that collected data on the perceptions of three user groups familiar with such a system: radiologists, radiology technologists, and medical specialists. The process yielded five main findings.

First, the assessments made of PACS vary according to the use made of the system. The scores for system quality and the quality of the data produced were markedly higher for intra-hospital use than for inter-hospital use. To our knowledge, no other study has measured levels of satisfaction for both types of use in a given population, but our results support the research in this field, insofar as the studies that have examined user satisfaction with intra-hospital use also found very high rates of satisfaction.4–6 Our findings are similar to those from a study conducted in an inter-hospital context that also reported a high level of dissatisfaction with response time.21 This problem has since been resolved through upgrades to the telecommunication network’s capacity. Second, this difference in assessments is also clearly seen in system use (only one respondent in four reported inter-hospital use). Our results from the satisfaction measure reveal that clinicians remain a demanding user group when it comes to the usefulness and ease of use of a technology. Scores for the variables Perceived usefulness and Ease of use were much higher for intra-hospital use than they were for inter-hospital use. This finding agrees with an earlier study of information technology adoption that identified these two factors as the main reasons explaining the adoption of new technologies.27,28 These two factors were even more critical in this case, since our sample of radiologists were under no obligation to analyze examinations made at the other sites. At the time of our survey, remote reading was still optional. Given the shortage of radiologists practicing in outlying regions, it was difficult to make this type of activity mandatory, even more so since the quality of the inter-hospital PACS was considered poor. This situation should improve with the deployment of a regional on-call medical imaging system. Third, it is important to stress that despite the negative assessments they made of inter-hospital use, users maintained a positive attitude toward some type of virtual organization of medical imaging. Indeed, the score for Overall satisfaction with the system was very high: 8.9 out of 10. Moreover, the scores for Intended future use of the system were very high for both intra-hospital use (8.9) and inter-hospital use (8.7). This is a particularly significant finding, given that the variable Confirmation of expectations was very high for intra-hospital use (8.0) but much lower for inter-hospital use (5.7). Fourth, we found significant differences in perceptions among user groups. These findings strongly resemble those reported by Paré et al.,25 who found greater satisfaction among radiologists and technologists than among medical specialists. However, our findings are different from those reported by Tually,21 who observed a higher level of satisfaction among radiologists than technologists, particularly in terms of system quality, data quality, service quality, and overall satisfaction. Fifth, our regression analyses confirmed that, as suggested by the DeLone and McLean model,22,23 there are indeed strong relationships between system quality, user satisfaction, and intended future use.

Before concluding, it is worth mentioning the limits of this study. This was first and foremost a cross-sectional analysis of data collected at a specific moment in time. In this context, our regression analyses measured associations and links between variables, but they cannot demonstrate true causality. In addition, our findings came from the analysis of a single virtual medical imaging organization which represented a first trial of this form of organization in Quebec’s health care system. It is important that this type of study be repeated in order to validate our findings and better understand this phenomenon.

In conclusion, we would like to reiterate the overall positive attitude expressed by the three main groups using hospital medical imaging. They were seasoned users who had the opportunity to fully test this new form of organization: inter-hospital PACS. Despite a certain amount of dissatisfaction that may be due to the innovative nature of this emerging form of virtual organization, we nevertheless found sustained user participation in the project, and the level of intended future use was high. This is encouraging news for the deployment and sustainability of this new form of virtual organization of medical imaging practices.