Introduction

The concept of user involvement in mental health care has been growing in popularity over the last 20 years, especially in countries that have transformed their previous institutionally based service provision into a community-oriented care model [1]. The “users movement” advocates a shift towards a modern “shared-” or “partnership model” between health professionals and service users: this model is based on mutual respect for each others’ skills and competencies and recognition of the advantages of combining these resources to achieve beneficial outcomes for patients and their families [2]. In addition, evolving from service user movements, the recovery model emphasises control being placed in the hands of mental health users and poses great emphasis on the collaborative nature of care between providers, consumers and their families [3].

Within this context, a ‘needs-led’ approach to mental health-care planning and outcome evaluation seems particularly useful, since it allows considering the multiple perspectives of those involved in the care process [4, 5]. In fact, “need” is a socially negotiated concept presenting no single “correct” perspective [6].

Previous studies [7–11] report that need perception differs considerably between professionals and patients and that disagreement between staff and patients frequently occurs on unmet needs. Whilst most of the studies have compared patient and staff perception of needs, only two studies have done so with patients and their caregivers [12, 13]. Previous research suggests that patients treated in community-based settings rate fewer needs than their caregivers [14] and that they also emphasise different needs [12, 15, 16]. Given the currently greater involvement of carers in mental health care, a better understanding of these differences has important service planning implications. Combining the three perspectives is however a puzzling task, since adding the family’s views on the patient’s needs makes the equation even more difficult to solve. Family members may hold different (and sometimes conflicting) values, goals and priorities with respect to both their ill relatives and treating staff. Nonetheless, we are convinced that the effort to combine the three perspectives is worth pursuing, since it may provide a more comprehensive and articulated view on how staff–users interaction works within “real-world” mental health services.

Another important user-perspective component of outcome is that of satisfaction with mental health services [17, 18]. Regular assessment of service satisfaction could help identify the strengths and weaknesses of a given service from the users’ perspective. Moreover, periodical monitoring of service satisfaction has proved useful in ascertaining service capacity for adaptation to organisational changes (i.e. resources, staff members’ personal and technical skills, cultural model shifts, administrative changes, etc.). Determinants of patients’ and family members’ satisfaction with services [19–22] have been extensively investigated, as have levels of service satisfaction, in several routine and experimental services [23–25]. Yet, data comparing patients’ and their family members’ perceptions of mental health care provided are still unavailable, although this approach could prove highly useful in complementing the assessment of mental health-care outcomes from multiple perspectives.

The present study aimed to compare the perceptions of staff members (SMs), patients and family members (FMs) concerning two important subjective outcome measures—needs for care and service satisfaction—in a sample of psychotic patients treated in either outpatient or day-care facilities. It was expected that comparison of these three groups’ perceptions would yield: a multifaceted picture of mental health outcome measures, different insights into the quality of care as perceived by the various stakeholders involved and a broad-based evaluation of the service’s strengths and weaknesses.

Methods

Design and participants

Our study was cross-sectional and the study group consisted of adult patients (age 18–60 years) with a DSM-IV diagnosis of non-affective psychosis (295.xx, 297.xx, 298.8, 298.9) who had received mental health care at the Legnago department of mental health (DMH)—the main NHS agency providing public psychiatric care for the adult population (nearly 140,000) inhabiting the southernmost part of the Verona province in Italy. Services involved in the study included a community mental health centre (CMHC) (which provides routine outpatient care to patients living in a sub-sector of the Legnago catchment area) and two day centres (which provide a daily programme of support, groups and rehabilitation activities for people with long-term mental health conditions). Patients were required to have received day centre or outpatient treatment for at least 12 months at the time of study inclusion (the 12-month limit was established to ensure sufficiently long contact to allow SMs to validly assess the patients’ needs). Other inclusion criteria were regular patient contact with mental health staff and at least one participating family member.

Overall, 116 patients (55 patients at the day centres and 61 at the CMHC) met the inclusion criteria. All were asked to participate in the study and were asked whether the researcher was allowed to contact their family members. Only one patient refused to participate, and the remaining accepted the involvement of their family members in the study. A total of 27 SMs and 120 FMs were included in the study.

The study began with patient interviews being administered by the same researcher. The patients, FMs and SMs were separately interviewed over a period of 1 week. Both patients and their family members were interviewed at the day centre and CMHC facilities, and staff members were interviewed in the same settings, but on separate occasions.

Measures

The Italian version of the Camberwell Assessment of Need (CAN) [26, 27] was used to assess needs for care; it is an interview developed for patients (CAN-P) and staff (CAN-S) comprising 22 items grouped into 5 conceptual domains: health (physical health, psychotic symptoms, drugs, alcohol, safety to self, safety to others, psychological distress), basic (accommodation, food, daytime activities), social (sexual expression, company, intimate relationships), service (information, telephone, transport, benefits) and functioning (basic education, money, childcare, self-care, looking after the home). Each item is assessed on a three-point scale: 0 = no problem, 1 = no/moderate problem due to continuing intervention (met need) and 2 = current serious problem regardless of whether help is offered or given (unmet need). The CAN-P surveys patients’ opinions on their own perceived needs for care; in a separate interview the same questions (CAN-S) are asked to patients’ key-workers. In the present study, the same CAN-S questions were also administered to the patients’ informal caregivers, so as to obtain information on patient needs from the three different perspectives of patients, SMs, and FMs. Test–retest and inter-rater reliability of the Italian version of the CAN was investigated in a sample of patients attending the South-Verona CMHS [27] and it was found to be good, also when used under routine conditions and without specially trained staff.

The brief version of the Verona Service Satisfaction Scale (VSSS) [28, 29] was used to assess satisfaction with mental health services. The brief VSSS consists of 32 items, which conceptually cover seven dimensions: ‘Overall satisfaction’, ‘Professionals’ skills and behaviour’, ‘Information’, ‘Access’, ‘Efficacy’, ‘Types of intervention’ and ‘Relative’s involvement’. The instrument is designed for self-administration and can be completed in 20–30 min. Respondents are asked to give overall ratings on their experience of the mental health services they have attended during the previous year. Satisfaction ratings for items 1–18 are listed on a five-point Likert scale (1 = terrible, 2 = mostly unsatisfactory, 3 = mixed, 4 = mostly satisfactory, 5 = excellent). The items are presented with alternate directionality to reduce the probability of stereotypic responses. Items 19–32 consist of three questions each: first, the subject is asked if he/she has received the specific intervention (Question A: “Did you receive the intervention x in the last year?”). If the answer is “yes”, he/she is asked his/her satisfaction on a 5 point Likert scale, as above (Question B). The VSSS, in its versions for patients and relatives, has been tested for acceptability, content validity, sensitivity and test–retest reliability in a sample of 75 patients and 75 relatives attending the South-Verona CMHS [28] and it was found to show good psychometric properties.

Statistical analyses

Comparisons between categorical variables were performed by Chi-square test or Fisher’s exact test if cell frequencies <5. Mean scores of continuous variables were compared among independent groups by t test (2 groups) or one-way ANOVA with Bonferroni correction (>2 groups). All tests were bilateral at p < 0.05. Total percentage agreement and Cohen’s weighted kappa [30] (using linear weights, based on ratings of ‘0’, ‘1’ or ‘2’ for each CAN area’s Section 1) were computed to yield a measure of agreement for the level of need. Percentage agreement is categorised into the following values: good ≥ 90%, adequate 80–90% and poor < 80% [31]. A kappa coefficient ≤ 0.20 indicates a slight agreement, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = substantial and 0.8–1.0 = almost perfect agreement [32]. The weighted index indicates the degree to which disagreements pertain to neighbouring categories. To avoid a discordance paradox between percentage agreement and kappa, marginal rating distributions were carefully examined [33]. Analyses were performed by SPSS 17.0 and Stata 9.2 for Windows.

Results

Sample characteristics

The patients’ mean age was 40.8 years (s.d. 9.6); 54 (46.6%) were women, and their mean illness duration was 10.0 years (s.d. 5.4). When interviewed, 110 patients reported living with their family of origin, and 10 were living alone in their own apartments. Patient sample characteristics are shown in Table 1.

Table 1 Sample socio-demographic and diagnostic characteristics by type of facility (n = 116)
Full size table

The family members’ mean age was 58.7 years (s.d. 10.2); 71 (59%) were women; the frequency of contact with patients ranged from every day to twice a week. The staff members interviewed had a mean age of 42.4 years (s.d. 6.1); 25 (92.5%) were women. The staff member subgroup had worked with psychiatric patients for a mean of 14.5 years (s.d. 5.3); all staff reported having been in contact with their patients for at least 12 months.

Patient, staff, and family member-rated needs

The sample reported the following numbers of needs: (1) Patients, mean total 5.78 (s.d. 3.48), i.e. 4.20 (s.d. 2.72) met and 1.59 (s.d. 1.60) unmet needs; (2) SMs, mean total 6.28 (s.d. 3.42), i.e. 4.14 (s.d. 2.29) met and 2.14 (s.d. 2.23) unmet needs; and (3) FMs, mean total 6.02 (s.d. 3.38), i.e. 4.11 (s.d. 2.48) met and 1.91 (s.d. 1.86) unmet needs. The mean differences among total, met and unmet needs were not significant.

Table 2 shows the mean number of total, met and unmet needs in the 22 CAN areas grouped into five main aggregated need domains.

Table 2 Patients’, staff members’ (SM) and family members’ (FM) aggregated ratings of met, unmet and total needs (n = 116)
Full size table

Significant differences were observed for the health domain, in which staff rated more unmet needs than patients (ANOVA Bonferroni, p = 0.014) and family members (ANOVA Bonferroni, p = 0.019). Moreover, the basic domain showed staff and family members as rating more unmet needs than patients (ANOVA Bonferroni, p = 0.038), as in the functioning domain, where staff and family members once again rated more unmet needs than patients (ANOVA Bonferroni, p = 0.033 and p = 0.045, respectively). No significant differences were found in the remaining aggregated domains.

Patients, staff and relative agreement on needs

The CAN areas most frequently (>50% of cases) identified by all three groups were: psychotic symptoms, company, psychological distress, daytime activities and looking after the home. The most problematic areas reported by the three groups were psychotic symptoms and company, although some between-group ranking differences emerged for the other areas. Patients most frequently reported needs in the areas of psychological distress (54.3%) and daytime activities (51.7%); staff most frequently identified needs in the areas of daytime activities (63.8%), psychological distress (62.1%) and looking after the home (55.2%). Lastly, family members most frequently reported the areas of daytime activities (59.5%), looking after the home (57.7%) and psychological distress (56%). Most patient–SM pairs (75.9%), patient–FM pairs (63.8%) and SM–FM pairs (69.0%) reported needs in the area of psychotic symptoms.

Table 3 shows patient–SM, patient–FM and SM–FM agreement levels for each CAN area.

Table 3 Total percentage agreement and weighted kappa coefficients (linear weights) based on ratings of 0, 1 and 2 of each individual area of the CAN for patient–staff member (SM), patient–family member (FM) and SM–FM pairs (n = 116)
Full size table

Percentage agreement coefficients in the various CAN areas (columns 2–4) followed a consistent pattern for the three groups: agreement was good (≥90%) in nine areas, adequate in eight areas (80–89%) and poor (<80%) in four areas Yet, the mean level of patient–FM pair agreement (87.8%) was somewhat lower than that observed for patient–SM and FM–SM pairs (88.6 and 88.9%, respectively).

To control for random agreement, a weighted kappa coefficient for patient–SM, patient–FM and SM–FM pairs was also computed for each CAN area (see Table 4, columns 5–7). Patient–SM agreement was almost perfect (>0.8) in 1 area, substantial (range 0.61–0.80) in 3 areas, moderate (range 0.41–0.60) in 11 areas and fair (range 0.2–0.4) in 7 areas. Patient–FM agreement was almost perfect (>0.8) in one area, substantial (range 0.61–0.80) in two areas, moderate (range 0.41–0.60) in ten areas and fair (range 0.2–0.4) in 9 areas. SM–FM agreement was substantial (range 0.61–0.80) in 7 areas, moderate (range 0.41–0.60) in 12 areas, fair (range 0.2–0.4) in 2 areas and slight (<0.20) in 1 area.

Table 4 Patients’ and family members’ mean scores of satisfaction with services (n = 116)
Full size table

Overall, the three groups showed a moderate mean level of agreement. Yet, similarly to percentage agreement, weighted kappa was lower for patient–FM pairs (0.45) than that for SM–FM pairs (0.54), with patient–SM pairs presenting an intermediate position (0.48). More discrepancies (fair–slight agreement) were observed in the patient–FM group (7 CAN areas), whereas the SM–FM group showed the highest degree of agreement on needs.

The greatest discrepancy in all three groups concerned “intimate relationships” and “sexual expression”, whereas the highest levels of agreement emerged for “telephone” and “transport”.

It should be noted that patient–SM pairs showed almost perfect agreement on “accommodation” (kappa = 0.45), but that both patient–FM and SM–FM pairs showed only moderate agreement in that area (0.43 and 0.49, respectively). Conversely, “safety to other” yielded substantial SM–FM agreement (kappa = 0.69), but patient–SM and patient–FM pairs showed poor agreement in that area (0.24 and 0.32, respectively).

Patient and family member-rated service satisfaction

Table 4 reports patient and FM mean satisfaction scores per VSSS dimension.

Overall, service satisfaction did not significantly differ for patients and relatives. The only VSSS dimension yielding a significant difference was “Information”, with patients reporting lower levels of satisfaction than their family members (p < 0.05; t test). It should be noted that the VSSS dimensions showing the lowest satisfaction scores for both patients and relatives were “Information”, “Access”, and “Type of intervention”.

Patient–family member agreement on service satisfaction

Table 5 shows patient–FM agreement for each VSSS item (both total percentage agreement and weighted kappa coefficients).

Table 5 Total percentage agreement and weighted kappa coefficients (linear weights) based on ratings of “0”, “1” or “2” for each individual area of the VSSS in patient–family member (FM) pairs (n = 116)
Full size table

Overall, both the percentage agreement and weighted kappa values showed poor patient–FM agreement across the various VSSS items: weighted kappa was slight (<0.2) for 21 items (65% of total VSSS items), fair (range 0.21–0.35) for 10 items (32% of VSSS items) and moderate (0.44) for 1 item only, i.e. “satisfaction with sheltered accommodation”. No VSSS item yielded substantial agreement.

Discussion

This study provides empirical evidence that disagreement among staff members’, patients’, and family members’ opinions represents the rule rather than the exception in mental health services, thus suggesting that their perceptions should be all considered when both planning interventions and assessing treatment outcomes. This process, however, is not unproblematic, particularly in care- planning. Sorting out the puzzle of the different views and combining them into a coherent and integrated therapeutic strategy, well accepted by all the parties involved in the care process, is not an easy task. Becoming aware of patterns of disagreement of staff, patient and family member about patients’ needs, however, is a necessary starting point from which service providers can work to increase consensus. This, within a partnership model of care delivery, represents a fundamental prerequisite to empower patients and to actively involve them.

Strengths and limitations

This is the first study to assess both needs for care and service satisfaction according to staff members’, patients’ and family members’ views in a sample of psychotic patients attending community-based psychiatric services. Previous research compared staff members’ and patients’ need perceptions [7–11]. Other studies conversely examined patients’ and family members’ perceptions [13], and only a few studies have compared the views of patients, staff and family members of needs for care [12, 14, 34, 35]. Most of the latter studies, however, were conducted in geriatric settings [14, 34, 35], not in mental health facilities serving the adult population. Moreover, the only study comparing the views of staff member’, patients and relatives [12] specifically focused on needs for care, without addressing service satisfaction.

This study also has some limitations. The main limitation is the cross-sectional design, which does not allow establishing causal relationships. Another limitation might be the ability to generalise the findings to other populations, such as first episode psychotic patients, given the predominant composition of long-term patients. In addition, there could be bias resulting from self-selection into the sample. Moreover, due to the limited sample size, we could not stratify the study population by the type of service (i.e. CMHC vs. day centre) to establish the potential occurrence of a “setting effect”

Comparing staff and patient perceptions of need

The present study confirms that significant discrepancies between staff and patient perceptions of needs do exist [7–11]. Intimate relationships, company, sexual expression and psychological distress were the CAN areas presenting the poorest staff–patient agreement. Staff and patients can disagree along these dimensions for various reasons. For example, lack of staff knowledge about a patient’s problems may impact on CAN areas such as sexual expression or intimate relations. Moreover, and more generally, varying socio-cultural, educational and professional backgrounds can lead to different views about what actually constitutes a problem for patients and may thereby apply to some CAN health-related areas.

The greatest degree of agreement was observed for CAN areas evaluated by staff and patients as presenting the fewest problems (e.g. telephone, transport, childcare, safety to others, basic education and money) and areas allowing for well-defined service responses, such as accommodation and physical health. These results are consistent with findings from previous studies [7, 8] showing good agreement only for areas in which specific service intervention had been provided and where patient problems had already been negotiated.

Comparing the perceptions of needs of staff and family members

Several discrepancies between staff members’ and family members’ perceptions were also detected. Yet, the average SM–FM agreement was substantially higher than both patient–SM and patient–FM agreement, suggesting thereby that staff and family members share more views on the nature of patients’ problems and what should be done to solve them. This is an encouraging finding, as it indicates that family member involvement with community mental health services is an achievable goal in the care process of individuals with psychoses. Families usually play a prominent role by helping supervise medication, encouraging patient participation in rehabilitation programmes and generally by providing an environment that can foster recovery or can at least reduce disability. Yet, endeavours to achieve optimal collaboration with families are frequently beset with a number of difficulties, including those of ethical nature (e.g. circumstances when the involvement of the family is justified despite a patient’s refusal to provide consent) [36]; mental health services should therefore make every effort to overcome these barriers by implementing specific programmes aimed at improving staff and family member communication and by supporting these families [37, 38].

Comparing patients’ and family members’ perceptions of patient needs and satisfaction

Patients and relatives presented only moderate agreement in most of the CAN areas; agreement was poor in the areas of help with intimate relationships, sexual expression and benefits. On average, patient–family member agreement on needs was substantially lower than that observed for patient–FM and SM–FM pairs. This finding points to the possibility of family members’ greater difficulty in understanding patient priorities and values.

Poor agreement between patients’ and family members’ perceptions of need have been reported elsewhere [12, 13, 34]. Modest agreement on needs might be attributable to family members’ perceptions of impaired patient judgement in these areas [39]. This finding, however, may also be due to the lack of a clear-cut definition of the need concept itself, or because patients and carers have different value judgements about the support patients require [34]. For example, patients place less emphasis on reducing symptoms and more emphasis on daytime activities, accommodation and social relationships [15] than family members typically do. Moreover, whereas relatives want more intensive support for patients, the latter tend to prefer a support style allowing them to maintain their independence [16].

Interestingly, systematic perception discrepancies were observed between patients and family members on most service satisfaction domains. Yet, both groups (especially patients) were most concerned about levels of information and advice received, facility layout and comfort, and type of interventions provided. Similar findings have been reported in several studies conducted in different settings [18, 23]. Indeed, lack of information appears to be a crucial determinant of dissatisfaction with psychiatric care for both patients and their relatives [22, 40]. Overall, these findings suggest that the caregivers of patients with psychosis seek rather basic assistance from psychiatric services and that they may serve as resources in caring for patients only if they receive sufficient support themselves, such as targeted psycho-educational interventions [41].

Conclusions

Specialised mental health services are currently struggling with the challenge of developing better methods to increase consensus on service needs and adequacy, while concurrently addressing resource limitations. This situation will require staff and service users to communicate more effectively in terms of their differing perceptions of patient needs, service options available and of their efficacy. Any effort to improve understanding among professionals, patients and relatives should be pursued, since it may positively impact on other treatment outcomes [42, 43]. It can be reasonably speculated that better staff–patient agreement can help users feel more responsible for their own treatment plans and may therefore result in an improved therapeutic alliance, greater treatment adherence, increased intervention uptake and, consequently, greater intervention effectiveness [44, 45]. However, any attempts to increase user consensus must go hand in hand with a reorientation in service treatment philosophy, i.e. by viewing users as ‘treatment team leaders’ and no longer as passive treatment recipients.