Introduction

Quality of care has become increasingly important in the evaluation of healthcare and healthcare services [1–3]. Evaluating rheumatic healthcare quality is a major issue given to the care need profile of patients with rheumatoid arthritis (RA) and their long-term dependency on healthcare [4]. Evaluation of quality of care is often performed by healthcare professionals. However, patients’ perspectives on healthcare quality differ from the views of healthcare professionals and policy makers [5–7]. Also, patients’ perspectives on the quality of care have become more prominent in research and policy since the introduction of the concept of patient-centered care in many countries [8, 9]. This concept aims to empower patients with respect to their healthcare decisions and to (re)structure the healthcare system according to their needs.

Patients’ views on quality of care have often been conceptualized as patient satisfaction [10–12]. A disadvantage of these surveys is that the scores are extremely subjective, highly skewed (>90% are satisfied), and influenced by personal preference and patient expectation [13]. Caregivers and healthcare services cannot influence patients’ expectations, but can change the actual experiences. Therefore, a more refined and less subjective instrument for evaluating healthcare quality from the patients’ perspective seems necessary. The Consumer Quality index (CQ-index) provides such an instrument [14, 15].

The CQ-index is based on two families of surveys that measures patients’ experiences. The first family of surveys that is used is the Consumer Assessment of Healthcare Providers and Systems (CAHPS®), which is well established and widely used in USA (www.cahps.ahrq.gov). This methodology comprises standardized protocols and manuals concerning sampling, data collection, data entry, data analysis and data reporting, which are also used as reference for CQ-index research. Furthermore, the lay out and answering categories on a four-point scale (never, sometimes, usually, always), three-point scale (not a problem, a small problem, a big problem), global ratings (ranging from 0 to 10, with a score of 10 indicating the best possible score) and referring to the frequency of experienced events in a certain time period (‘In the last 12 months, how often…’) were taken from the CAHPS. The second family of instruments that is used is QUOTE (QUality Of care Through the patient’s Eyes), which has been developed in the Netherlands [16–20]. From this methodology the questions about importance (answering categories ‘not important’, ‘fairly important’, ‘important’ and ‘extremely important’) were added into the CQ-index methodology. Also, the concept to include disease-specific items of healthcare services is derived from the QUOTE instruments. The combination of these survey features has an outcome that every CQ-index exists as two parts: one about patients’ experiences and one about the important patients’ award to quality aspects.

Within rapid development of CQ-index instruments (up to 25), priority has been given to the development of a new instrument for patients with RA. The CQ-index RA is a disease-specific survey, designed to assess patients’ experiences and importance on quality aspects. Patients with RA are ‘heavy healthcare users’, most of them use two types of healthcare services besides the rheumatologist [4]. For patients to decide which specific provider to choose, consumer information about the quality of care of these providers can be helpful. Measuring patient experiences enables us to provide such information. In addition, the CQ-index allows professionals to compare the quality of care they provide with colleagues in the same profession. This instrument may also be an important tool for healthcare providers to monitor their quality of care. Finally, the CQ-index information identifies elements of care that, according to patients, are particularly eligible for improvement.

The aim of this article is to provide an instrument to measure patients’ experiences about the healthcare. For this purpose, information on the content and psychometric characteristics of the developed CQ-index RA, as well as on the importance patients assign to the different quality aspects are presented.

Patients and methods

The creation of the CQ-index rheumatoid arthritis questionnaire

The survey was formed following the protocols of the CQ-index standard, which is based on the CAHPS and the QUOTE [15, 21]. A draft questionnaire was constructed using input from two focus group discussions with in total 22 patients with RA, existing questionnaires measuring patients’ experiences with quality of rheumatic care (i.e. QUOTE-Rheumatic patients [20]) and professional guidelines for RA [developed by the Dutch Institute for Healthcare Improvement (CBO)]. The draft questionnaire was presented to an expert panel (health insurance companies, a patient organization, healthcare providers, patients with rheumatoid arthritis) to assemble comments and improve the instrument. Their comments were included in the second draft of the questionnaire, which was used, after cognitive testing, to test its psychometric properties.

CQ-index RA experience

The pilot instrument consisted of two parts, the CQ-index RA experience and the CQ-index RA importance. The first part, the CQ-index RA experience, contained 15 items on person characteristics (e.g. age, education, patient’s self-reported physical and psychological health), 6 global ratings (of caregiver during control of care, professional who gives medication, specialized nurse, therapist, medical specialist who performs the surgical procedure, rheumatic care in general), and 114 experienced items regarding the actual experiences of patients with specific quality aspects of care. Other items were instruction items or items with answering categories other than the standard format. The items were divided in 16 themes (Table 1).

Table 1 Content CQ-index RA experience (number of questions)
Full size table

The answering formats of the experience items were: (1) never, sometimes, usually, and always, (2) not a problem, a small problem, and a big problem, and (3) yes and no. The global ratings ranged from 0 to 10, with a score of 10 indicating the best possible score. If questions were not applicable, an answering option was added. The grade of education was divided into ‘no education and secondary education’ and ‘higher than secondary education’. The self-reported physical and psychological health was assessed using a five-point scale: ‘excellent’, ‘very good’, ‘good’, ‘fair’ and ‘poor’. The instrument instructed respondents to reflect on their experiences in the last 12 months.

CQ-index RA importance

In the second part, the CQ-index RA importance, 69 importance items were included, which were designed to assess the importance that patients attribute to aspects of healthcare, using a four-point scale: ‘not important’, ‘fairly important’, ‘important’ and ‘extremely important’. The importance survey contained fewer items because quality aspects such as ‘to be taken seriously’ applied to healthcare professionals in general rather than each healthcare professional of a particular discipline.

Subjects

A total of 590 patients with RA were selected from the files of four health insurance companies. Inclusion criteria were: (1) being older than 18 years of age, (2) having received rheumatic care in the last 12 months (based on declared costs) and (3) not being approached in the past for other CQ-index surveys. Selecting patients from the files of health insurance companies provided us with a random selection of the RA population, which was heterogeneous in treatments they received.

Data collection

This study was conducted in the Netherlands in the summer of 2007. All selected patients received a mail from their health insurance company with the request to fill out the CQ-index RA experience and CQ-index RA importance. A “thank you” card was sent to all patients 1 week later. Non-respondents received a second survey and letter in the fifth week, and a reminder letter in the seventh week. The mailing packs with the questionnaires included a stamped addressed envelope [22].

Analyses

Response

The response rate was calculated after excluding the patients who were not willing or able to participate, replied double, or responded negatively to the question whether they had received rheumatic care in the last 12 months. Furthermore, patients who stated that they did not answer the questions themselves and who filled out less then half of the core items (items that should be answered by all respondents) were also excluded [21, 23].

Subjects

To determine whether the respondents and non-respondents differ they were compared on age (t test) and sex (χ 2 test).

Dimension structure

Conducting one explorative factor analysis with a direct oblimin rotation was not possible, because not all items applied to all patients. Therefore, several explorative factor analyses (EFA) with a direct oblimin rotation were conducted on groups of different items (themes) of the experience survey to examine the structure of the questionnaire and the relationship between items with similar answering categories. The factor analyses were accurate if the Kaiser–Meyer–Olkin Measure of Sampling Adequacy (KMO) was higher than 0.60. The number of factors per factor analyze was determined by Kaiser’s criterion (Eigenvalues > 1.00). If the factor loading of an item exceeded the criterion of 0.40, the item was assigned to that particular factor. If an item loaded across multiple factors, it was assigned to the factor where it had the highest loading.

Reliability

The internal consistency reliability of the scales from the experience data was estimated using Cronbach’s alpha, where an alpha value of 0.70 or more was considered satisfactory [24]. Item-total correlations (ITC) were also calculated, to correct for item overlap and check for homogeneity of the simple-summated scales, which were created from items that loaded strongly on the factors(criterion > 0.4 ITC) [25]. For one module (first visit rheumatologist), the reliability of a scale was calculated by the Kuder–Richardson Formula 20 (KR-20), which is the alternative method to assess the reliability for dichotomous items (yes/no) [26].

Importance

The importance scores were calculated to determine the mean importance of each quality aspect (not important = 1, important = 2, fairly important = 3, extremely important = 4).

Quality improvement score

This score represents the proportion of respondents who have had a negative experience (‘never/sometimes’, ‘no’ or ‘a big problem’) on a quality aspect multiplied by the importance score of the same quality aspect (range 1 ‘not important’ to 4 ‘extremely important’). The quality improvement score is a score that gives healthcare providers information about which quality aspects are particularly eligible for improvement; the higher the score, the more potential there is for improvement.

Analyses were performed using SPSS 14.0®.

Results

Response

At the end of the data collection period, 472 patients returned the CQ-index RA experience and 445 patients returned the CQ-index RA importance. After exclusion, the experience data of 407 patients (69%) and the importance data of 385 patients (65%) were analyzed.

Subjects

The majority of the respondents were female (72.7%), and had an education on a secondary or lower level (50%). The self-reported physical health was moderate (47%) or good (41%), and the self-reported psychological health was mostly good (61%) followed by very good (12%) and excellent (12%). Respondents and non-respondents differed from each other concerning age (mean age was 62.9 vs. 59.7 years; t = 2.86; p = 0.004), but not concerning gender (χ 2 = 1.76, p = 0.18).

Dimensional structure experience survey

Eight EFA were performed for different themes of the CQ-index RA Experience survey (themes 4–10, 13; see Table 1). For one explorative factor analysis (theme 13 ‘Financial compensation’) the KMO value was not sufficient (KMO = 0.50). The primary factor loadings based on the pattern matrix are presented in Table 2. In total, eleven factors were formed: (1) Conduct caregiver during control of care, (2) Competence caregiver during control of care, (3) Remaining items control of care, (4) Competence caregiver medication, (5) Conduct specialized nurse, (6) Competence specialized nurse, (7) Conduct therapist, (8) Remaining items therapist, (9) Conduct psychological care, (10) Conduct occupational physician, (11) Cooperation, and (12) Remaining items cooperation. With the exception of question 35 (Q35, factor 1, α = 0.38), all items used in the scales exhibited factor loadings exceeding 0.40. Q35 was removed from factor 1. All items that did not fit in a scale were reported separately (not presented in this article).

Table 2 Factor loadings of the items in the CQ-index experience according to the eight explorative factor analysis with oblimin rotation (EFA1–EFA8)
Full size table

Reliability

The fourth column (α1) of Table 2 shows the Cronbach’s alpha coefficients for the 11 formed factors, which ranged from 0.00 to 0.94. Three factors had poor reliability resulting in Cronbach’s alpha coefficients of 0.00, 0.37 and 0.39. Removal of any of the items did not increase the alpha coefficient to the threshold of 0.70 (see column 5 of Table 2), and therefore, we were not able to create reliable scales with these items. One factor displayed a relatively low reliability (factor 6; α = 0.67), after removing Q50 from this factor, Cronbach’s alpha increased to 0.86. The internal consistency was tested for a thirteenth factor consisting of seven dichotomous items. In this factor Q17 had a low ITC, which showed that this question does not fit well in this factor. Excluding this item resulted in a KR-20 coefficient of 0.86, indicating a good reliability for factor 13 ‘conduct rheumatologist’. Regarding the other formed scales (except for the scales with a poor reliability), the ITC was adequate. In total, ten reliable scales ranging from 0.77 to 0.94 were constructed (factors 1–2, 4–7, 9–11, 13; see Table 2).

Importance scores

Table 3 presents the ten most important quality aspects mentioned by the respondents. Patients rate the item ‘caregivers consider other medication when prescribing drugs’ as the most important quality aspect (importance score = 3.7). Information about the risks and the long-term consequences of a surgical procedure, the process of the surgical procedure and the rules for what (not) to do after a surgical procedure are important (3.5–3.6). The rapid accessibility and availability of care (3.5–3.6) are also valued highly. Finally, caregivers have to take patients seriously (3.5) and patients need to discuss arrangements about what to do when RA deteriorates (3.5). Items that did not fit a scale were candidates to be removed from the questionnaire (reliability analysis). Other arguments for adjusting this instrument were based on the dimensional structure, importance scores, and comments of expert panel. For more details see rapport of Zuidgeest et al. [27].

Table 3 The ten highest importance scores
Full size table

Quality improvement score

In Table 4, ten quality aspects with improvement scores are presented. A score of 2.11 is the highest quality improvement score for the quality aspect ‘information on a website of the hospital about RA’. ‘Information about reimbursements and contributions about support and devices’ (Q = 1.69) and ‘general practitioner asked about having rigid joints’ (Q = 1.61) were second and third in this overview. Other aspects concerned ‘reporting of a mistake’ (Q = 1.59), ‘more information about adjustments at home’ (Q = 1.51), ‘support concerning domestic activities’ (Q = 1.44) and ‘personal care’ (Q = 1.35), and ‘talking with caregivers if something went wrong’ (Q = 1.34).

Table 4 Top ten quality improvement scores (Q) with corresponding importance scores (I) and experience score, as proportion negative experience (E) are displayed
Full size table

Discussion

The aim of this study was to illustrate that patients’ experiences can be reliably measured with the CQ-index and therefore data collected with this instrument can be an important source to evaluate quality of care. The content, the psychometric characteristics of the CQ-index instrument for patients with RA, and patients’ views on the quality of rheumatic care were investigated. Our findings show that the multidisciplinary provides us information regarding experiences of patients with RA with several healthcare professionals (rheumatologist, specialized nurse, therapist, occupational physician, medical specialist, psychological caregiver). The final version of the questionnaire consists of ten scales with good reliability ranging from 0.77 to 0.94. Analysis revealed that the utmost important issue according to patients’ was alertness when prescribing drugs. Providing information on a special website of the hospital about RA was the quality aspect that needs the most consideration, as appeared from the quality improvement score.

Measuring patient’s experiences to evaluate quality of care is relatively new and goes beyond measuring patient satisfaction, which is a multidimensional concept, influenced by personal preferences and patient expectations. By separating the dimensions ‘experience’ and ‘importance’ individual providers and healthcare organizations get a clearer view on the quality aspects that are susceptible for improvement; where the actual care does not meet the needs and expectations of patients. Patients’ experiences are useful as an extra source for quality assessment, next to clinical indicators and methods.

The fact that this new instrument represents a multidisciplinary survey is rather a unique feature, as most research predominantly focuses on a particular discipline, e.g. rheumatology [28]. Jacobi et al. [29] was the first who discussed research of multidisciplinary healthcare utilization among patients with RA with different caregivers (general practitioner, medical specialists, allied healthcare, home care). They reported inadequate quality in the field of rheumatic expertise, particularly for general practitioners, physiotherapists, home nurses and home help, and in the field of information on medication and treatment for rheumatologists and general practitioners. We were able to partly replicate this finding in our study. We also found inadequate quality on the aspect ‘giving information about home adjustments’ and on aspects regarding information topics. In our study, home nurses and home help were not evaluated.

Our finding that alertness when prescribing drugs is the utmost important aspect according to patients is not very surprising, considering the average daily intake of two doses of drugs [30]. The ten displayed quality aspects differed little (importance score between 3.49 and 3.65), which showed that patients with RA regard these quality aspects in general as ‘important’ to ‘extremely important’. The aspect with the highest possibility for improvement was providing of information on a special website of the hospital. A study of Culver and Chadwick [31] revealed that the Internet is a poor source of English information for patients with RA. Quality information was scarce and finding information was time-consuming. Hospitals may anticipate on this by providing their own information about RA as service to their patients.

Though the CQ-index RA was a relatively long questionnaire, the response rate was 69%. This is consistent with other disease-specific surveys [16]. The fact that three reminders were sent may have contributed to the relatively high response [32]. It is known that elderly women are more likely to return the questionnaire than young men [14]. In our study, we also found that older patients returned the questionnaire more often than younger patients, there was, however, no difference between male and female respondents. This could have had an effect on the results, because older people generally report more positive experiences than younger people [33, 34]. In other words, the experiences of patients could be biased in a positive direction.

CQ-index instruments are based on a multiphase development process, starting with focus group discussions and ending with large quantitative testing. In this large quantitative testing, relevant case mix adjusters are selected and used in the instruments developed so far [35]. Also other CQ-index instruments have revealed good ability to measure differences between healthcare providers, so called discriminative power [36, 37]. Information derived from CQ-index instruments was successfully translated in public choice information on a special website (www.kiesbeter.nl).

Because the CQ-index method is standardized in measuring consumer experiences, a big advantage is that quality information can be compared between different settings and/or different healthcare sectors. This contributes to the increasing transparency and performance providing of different healthcare providers. Providers can look to each others performance and adopt, where necessary, initiatives that have been used successfully in the past.

In conclusion, it is possible to reliably measure patients’ experiences with the quality of rheumatic care. The CQ-index RA is able to identify those aspects of care that require extra consideration. In addition, the instrument can be used in further surveillance tools in evaluating performance of different healthcare providers. Therefore, in a next phase, the CQ-index RA will be adjusted and tested on its ability to discriminate between healthcare providers on the constructed scales and quality aspects.