Introduction

Mental health problems are a leading cause of morbidity and impairment in social and occupational functioning [1, 2]. Depression is currently ranked third in the WHO Global Burden of Disease rating system and by 2030 is projected to be the leading worldwide cause of burden [3]. Impairment of functioning is not limited to those with a diagnosis of depression and a substantial proportion of people with anxiety disorders or psychoses (or indeed other psychiatric disorders) are unable to work or find work very difficult. There is evidence that many want to work but are unable to find suitable employment or adequate support to maintain it [4, 5]. Impairments in social functioning are increasingly being monitored in clinical practice and interventions are being targeted at them [6, 7]. At the same time alterations in social functioning are used as important outcome measures in controlled studies of novel service configurations and pharmaceutical agents [8–10].

The International Classification of Functioning, Disability, and Health (ICF) [11] was endorsed in 2001 by the WHO general assembly and complements the International Classification of Disease version ten (ICD-10) [12]. Its stated aim was to ‘provide a unified and standard language and framework for the description and classification of health and health-related states’. In the ICF there is a clear distinction between disorders of function (i.e. illness signs and symptoms) and disorders of capacity (limitations in actually executing activities), with no direct relationship between the two. Disorders of function and disorders of capacity can directly or indirectly lead to disorders of participation such as sick leave (an inability to perform at work). The ICF is a complex document and the Mini-ICF-APP (Mini-ICF-Rating for limitations of Activities and Participation in Psychological disorders) was developed to allow the reliable and rapid measurement and recording of impairments of capacity.

The Mini-ICF-APP was developed in Germany and there are now also English and Italian versions. It has been fully validated [13, 28] and is in widespread use in Germany. The German validation study demonstrated significant correlations with validated measures of functioning and illness severity but only partial correlations with duration of sick leave. This study will assess the English language version of the Mini-ICF-APP in a community sample of patients with mental health problems under the care of a Community Mental Health Team (CMHT). The original German version of the observer-rated instrument was translated into English by its originator ML before discussion with English speaking colleagues AM, TB, KY, and CA. In the United Kingdom, CMHTs provide support to the majority of people with severe and enduring mental health problems such as psychoses and severe mood or anxiety disorders. We chose this population as occupational difficulties are common with high rates of unemployment and long-term sick leave. Indeed, specific vocational services have been set up in most areas to intervene and improve outcomes for this group.

Aims of the study

To determine the validity and reliability of the English language version of the Mini-ICF-APP in a community sample of patients with mental health problems under the care of a Community Mental Health Team (CMHT).

Methods

Sample

A convenience sample of 105 patients in total was recruited from two Community Mental Health Teams in Oxford Health NHS Foundation Trust between February and September 2011. The sample size calculation was based on the correlation between the German Mini-ICF-APP total score and sick leave reported as r = 0.36, p < 0.001. It was found that n = 90 participants would provide an 83 % power to detect a similar correlation at the 1 % significance level (and a power of 60 % at the 0.1 % significance level). Similar calculations determined that a sample size of 47 would provide an 80 % power to detect a significant result at the 5 % significance level.

The sample was stratified to include a minimum of 30 patients each with a primary diagnosis of schizophrenia (n = 33), anxiety disorders (n = 30), and depression (n = 42). The only exclusion criteria were an inability to communicate in English or that the individual was of retired status. The protocol was approved by the Portsmouth and Isle of Wight Research Ethics Committee on 21/01/2011 with the following reference attached: 10/H0501/65.

Design

This was a cross-sectional study. All 105 participants were interviewed on one occasion by a trained researcher who administered the instruments described below and collected demographic and clinical information from medical records. Researchers were formally trained in the use of all instruments prior to commencement of the study. Patients were identified and approached through their individual clinicians in the mental health teams involved in the study. Informed consent was sought and recorded in all instances prior to interview. Forty-seven participants were rated simultaneously by another trained researcher to establish the inter-rater reliability and 50 participants were interviewed on one further occasion after 4 weeks to establish the test–retest reliability. Participants were compensated for their time with £15 at each interview. Interviews lasted for approximately 45 min.

Instruments

The Mini-ICF-APP [13, 14] was developed to measure limitations of capacity in the context of mental disorders. It follows the structure and dimensions of the ICF [11], building upon the definitions of the Groningen Social Disabilities Schedule II [15, 16]. The instrument is rated by the interviewer and the usual time scale for rating is the past 2 weeks. Thirteen domains of capacity are assessed: (1) adherence to regulations, (2) planning and structuring of tasks, (3) flexibility, (4) competency, (5) endurance, (6) assertiveness, (7) contact with others, (8) group integration, (9) intimate relationships, (10) non-work activities, (11) self care, (12) mobility, and (13) competence to judge and decide. Each dimension is rated on a five-point Likert-scale (0 = no impairment, 1 = mild impairment, 2 = moderate disability, 3 = severe disability, 4 = total disability). Anchor definitions for each item are provided in the rating manual. Rating uses all available information including the individual’s self report, case record, and observations from the interview situation. The total score can range from 0 to 52.

The Endicott Work Productivity Scale (EWPS) [17] is a self-report measure of performance at work. It records functioning at the workplace on a behavioural level in a detailed and specific way (e.g. “Arrive at work late or leave work early”, “Have trouble organizing work or setting priorities”). The EWPS consists of 25 items, each rated on a 5-point scale (from 0 = never; 4 = almost always). The total score ranges from 0 (no impairment) to 100.

The Brief Psychiatric Rating Scale (BPRS) [18, 19] is an interviewer-rated measure of severity of psychiatric symptoms. The 24 items of the BPRS are rated from 1 (no pathology) to 7 (extreme severity). The total score ranges from 24 to a maximum score of 168, with higher scores indicating higher levels of symptoms.

The Social and Occupational Functioning Assessment Scale (SOFAS) [20] is derived from the Global Assessment of Functioning Scale [21] and is administered by a trained interviewer. It is a well-established and validated measure of social and occupational functioning [22]. Scores range between 0 and 100, with higher scores signifying higher levels of functioning.

The Personal and Social Performance Scale (PSP) [23, 24] is derived from the SOFAS and is an interviewer-rated scale of broad social functioning. Levels of functioning are rated in four different domains: socially useful activities, personal and social relationships, self-care, disturbing and aggressive behaviours. These are then combined using a simple algorithm to arrive at a final score between 0 and 100, with higher scores indicating higher overall levels of functioning.

The Social Outcomes Index or SIX [25] is a brief ranking index that captures basic information in three domains: employment, accommodation, and social engagement. It takes only moments to complete and scores range between 0 and 6, with higher scores indicating better functioning.

Sociodemographic details collected included age, gender, ethnicity, marital status, employment status, and length of sick leave.

Statistical analysis

Descriptive statistics of sociodemographic data included the mean and standard deviation for normally distributed data, median and interquartile range for non-normally distributed data, and number and proportion for categorical data. Comparisons across diagnostic groups were performed through an ANOVA test for normally distributed data, Kruskal–Wallis for non-normally distributed data, and a Chi-square test for categorical data. Post hoc tests were carried out wherever an overall statistically significant difference was observed to further investigate the differences.

The instrument’s dimensionality was evaluated using the following methods: The internal consistency of the Mini-ICF-APP was assessed using Chronbach’s alpha. Shrout’s [26] suggested classification was followed for output interpretation. Response rates for the complete instrument were calculated and item response proportions and item-total Spearman’s correlations were computed to assess the functioning of the individual items. Principal component factor analysis (with a Varimax rotation method) was used to determine whether the Mini ICF-APP was a uni-dimensional or multidimensional questionnaire.

The stability (or reproducibility) of the instrument was evaluated by means of the inter-rater and test–retest reliability coefficients. Both were obtained by calculating the intra-class correlation coefficient (of two consecutive measurements by the same rater and of two simultaneous measurements by two different raters, respectively) using a two-way random model and absolute agreement in SPSS v.20. Shrout’s classification was again followed for output interpretation.

The construct validity of the Mini-ICF-APP was established by assessing the association of the total sum score with employment status (as measured by currently having a job) and length of sick leave (categorised into four groups depending on length of time off) in those who were employed. An independent samples t test was performed for the association with employment status and an ANOVA test for length of sick leave. The convergent validity of the Mini-ICF-APP was determined by calculating Pearson correlation coefficients for the total sum score and each of the illness severity and social functioning scales: EWPS, PSP, SOFAS, BPRS, and SIX score.

Simple linear regressions and the Bland–Altman graphical method for assessing agreement between two clinical measurements [27] were carried out as confirmatory analyses for both reliability and validity.

Results

Sample characteristics

The sociodemographic and clinical characteristics of the whole sample and each of the three diagnostic subgroups are presented in Table 1. Those in the psychosis group had spent the longest time on state benefits (p = 0.05), were less likely to be in work (p = 0.29), and more likely to be single (p = 0.02). They were less likely to have children (p = 0.003) than the other two groups and more likely to live alone (p = 0.003). Mean scores on the SIX and EWPS suggested those in the psychosis group may have been more socially and occupationally impaired but there were no significant mean differences on other measures.

Table 1 Sociodemographic characteristics of the Mini-ICF-APP sample
Full size table

Dimensionality of the Mini-ICF-APP

All Mini-ICF-APP items had a 100 % response rate. Item responses fell mostly between ‘no impairment’ and ‘moderate disability’ with an average mean item response of 1.13 (SD 1.13). Our analysis showed that the Mini-ICF-APP has substantial internal consistency for both time points and raters. Values of Chronbach’s alpha were 0.869 and 0.859 for raters one and two, respectively, at time point one and 0.912 for rater one at time point two. All 13 items on the instrument were significantly correlated at the p < 0.001 or p < 0.01 level with the total score on the Mini-ICF-APP, for both time points and both raters.

Reliability of the Mini-ICF-APP

Test–retest reliability returned a single measure intra-class correlation coefficient of 0.832 for the Mini-ICF-APP total score. Linear regression revealed a coefficient of 1.03 with p < 0.001, and adjusted R 2 = 0.91, supporting the substantial reliability found. Inter-rater reliability returned a single measure intra-class correlation coefficient of 0.886 and linear regression revealed a coefficient of 0.94 with p < 0.001, and adjusted R 2 = 0.93, supporting the substantial levels of reliability observed. Figure 1 illustrates the test–retest and inter-rater reliability of the Mini-ICF-APP (English version).

Fig. 1
figure 1

Scatter plots demonstrating test–retest (left) and inter-rater reliability (right)

Full size image

Validity of the Mini-ICF-APP

Table 2 presents the Pearson correlation coefficients between the Mini-ICF-APP items and total score and the other rating scales. There are strong positive correlations between the Mini-ICF-APP sum score and the sum scores on the EWPS and the BPRS (r = 0.7, p < 0.001, for both). There are strong negative correlations with the PSP sum score and the SOFAS sum score (r = −0.8, p < 0.001, for PSP; r = −0.7, p < 0.001, for SOFAS) as would be expected as in these scales a higher sum score represents better functioning. There is a weaker but significant negative correlation with the SIX sum score (r = −0.4, p < 0.001). The corresponding Bland–Altman plots also showed good agreement between the Mini ICF-APP total score and all indicators after standardisation of measurement.

Table 2 Correlations of the Mini-ICF-APP with established measures of illness severity and functioning
Full size table

Analysis of the relationship between Mini-ICF-APP sum score and employment status using t test showed a highly significant positive association at the p < 0.001 level. The mean sum score was 18.4 (SD 9.1) in the unemployed group and 9.4 (SD 6.4) in those who were employed. We found no significant association between the Mini-ICF-APP sum score and duration of sick leave. These findings are presented in Table 3.

Table 3 Total Mini-ICF-APP score by employment status and length of sick leave
Full size table

Discussion

Mental health problems are a leading cause of social and occupational disability [28]. Attempts to assist individuals with mental health problems to maximise their overall functioning and to return to or maintain work are an increasingly important part of health and social care interventions. The first step is the measurement of impairment, both in terms of its nature and severity. The publication of the ICF and subsequent development of the Mini-ICF-APP in German allowed for the rating of an individual’s level of impairment (disorders of capacity) in a number of different domains that could affect their ability to work (disorders of participation). This instrument has been validated in German and extensively used in healthcare systems in Germany. The aim of this study was to establish its reliability and validity in English as well as its utility in a large sample of general psychiatry community patients.

The Mini-ICF-APP demonstrated substantial inter rater and test–retest reliability as measured by intraclass correlation coefficients of 0.886 and 0.823, respectively. These values signify substantial correlations and were both highly significant at the p < 0.001 level. The instrument also demonstrated substantial internal consistency as measured by Chronbach’s alpha and explained in detail above. Our analysis suggested that there may be four groupings of domains within the overall instrument that are more highly correlated with each other. This may allow for further refinement of the Mini-ICF-APP and use of sub scores to allow more sensitive rating of impairments. Further analysis of this issue is the subject of a further publication.

Validity has been established by the strong and statistically significant correlations between the Mini-ICF-APP sum score and a number of well-validated measures of functioning and of psychiatric symptoms: the BPRS, EWPS, SOFAS, and PSP. There was a highly significant association between sum score and employment status. We found no evidence of an association between Mini-ICF-APP sum score and duration of sick leave. Linden and colleagues [29] had previously found a partial correlation of r = 0.34 between these variables and concluded that sick leave was a ‘crude but important’ measure for disorders of participation. Our results may lead one to conclude that it is a less useful measure in the population we studied in the UK and that other factors such as financial implications (the ‘benefit trap’), social expectations, and stigma are of more significance than objectively measured ability to complete tasks.

In summary, detailed statistical analysis has demonstrated that the Mini-ICF-APP is a reliable and valid measure of disorders of capacity in the terms of the International Classification of Functioning. We have demonstrated that it is possible to train staff and use the scale in large numbers of community mental health care patients in a UK setting.

Our results support the use of the instrument more widely in this patient group, both in clinical and research settings. Further consideration is required as to whether there are recognisable and useful sub scales that could allow the use of the instrument to be more sensitive than the current use of the sum score only. Further work would also benefit from including measures of discriminant validity as this was not included in this study.

Further work is needed to ascertain whether clinical staff can be trained and motivated to routinely use the Mini-ICF-APP in clinical practice and to target occupational interventions more precisely in terms of nature of intervention and of recipient. Although there are a number of measures of wider ‘social functioning’, there are few measures specifically of occupational functioning that have been validated and used in clinical practice. Given the increasing focus upon occupational outcomes in those with severe mental health problems and the relative lack of alternative measures, we propose that the Mini ICF-APP has the potential to be a useful and commonly used measure in clinical services and the evaluation of interventions.