Introduction

The KIDSCREEN-27 questionnaire, designed by Ravens-Sieberer, Gosch, Erhart and colleagues in 2006 [5], has been used to measure health-related quality of life (HRQoL) among children and adolescents in a number of countries and in a variety of settings including at home and in school [1–4]. The KIDSCREEN-27 has been shown to be a reliable and valid measure of HRQoL in children and adolescents [3], and substantial evidence for its cross-cultural equivalence and factor structure has been found [4].

The usual method of administration of the KIDSCREEN-27 is a paper questionnaire completed by the children and adolescents themselves or by a proxy such as a parent or teacher. According to the KIDSCREEN handbook [5], a computer version of the KIDSCREEN-27 is available but, to date, none of the studies that have been reported in the literature have used this mode of administering the questionnaire. The study on which this paper is based used the KIDSCREEN-27 embedded within an animated Internet survey of Primary 7 (P7—seventh grade of primary school) children in Northern Ireland in 2008. The main aim of this paper is to examine the factor structure and internal consistency reliability of the data obtained from a computer version of the KIDSCREEN-27 questionnaire.

What is Kids’ Life and Times (KLT)?

KLT is an annual Internet survey of all P7 (aged 10 and 11 years) children in Northern Ireland and is carried out by ARK that is a joint initiative between Queen’s University Belfast and the University of Ulster. The survey began in 2008. P7 children are in their final year of primary school, and the vast majority will be making the transition to secondary-level education at this age. As a consequence, this is an important stage in children’s development, and the survey is an affirmation of the child’s capacity to reflect on his or her life and school years to date, and recognition of the importance of doing so. The overall aim of KLT is to provide a vehicle for children to express their views on a range of policy-relevant issues such as bullying, education and health and well-being. The pilot KLT, which was carried out in 2008, included the KIDSCREEN-27 to measure the children’s HRQoL.

Internet surveys

There are a number of advantages of using the Internet to collect survey data from children. First, it is a medium that they are familiar with and enjoy using [6]. Secondly, the Internet allows the incorporation of graphics and sound within the survey to make it as much fun and as interesting for the children to complete as possible [7]. Thirdly, the cost of Internet surveys, relative to other modes of questionnaire administration such as face-to-face interviews, is low [8]. Finally, if the survey is designed well, the quality of the data is excellent; there are few missing values and little validation is required [8, 9]. Disadvantages associated with using the Internet to collect survey data have been identified in the research literature including biased samples [7, 10, 11], system incompatibility [12] and lower response rates compared with other methods of data collection [13–15].

According to De Vaus [7], bias arises mainly from the fact that not everyone has access to the Internet and those that do are not representative of the population in general. However, given that all primary schools in Northern Ireland have at least one computer with an Internet link, no child within the target population of children in P7 would be excluded on the basis that he/she could not access the questionnaire. Cognizant of possible biases based on what Hartford, Carey and Mendonca [12] called ‘system incompatibility’, the survey design team worked with Classroom 2000 (C2K), which is the organisation responsible for the provision of an information and communication technologies-managed service to all schools in Northern Ireland. The system used to deliver the questionnaire was compatible with the software available in the schools. To try to ensure a good response rate, principals were offered a confidential report for their school including comparison data with the Education and Library Board in which they were based and with the Northern Ireland sample as a whole. In these ways, the KLT team sought to exploit the advantages of using the Internet method of data collection with children while addressing some of the recognised drawbacks associated with its use.

Mode of administration

It has been claimed that the mode of administration of a questionnaire can affect the findings of a survey [16]. According to Hayslett and Wildemuth [17], mode effects are the effects of the medium of survey administration on the specific answers given by the respondents and, therefore, the results of the survey. Findings from studies of mode effects are mixed: some studies have identified interactions between the survey medium and the participants’ responses [18] while others have not [19, 20]. The increasing availability of the Internet and the relatively low cost of online surveys have made this an attractive mode of data collection to many researchers, and comparisons between Internet surveys and other modes of administration have begun to predominate in the research literature [18–22]. Several of these studies have considered the impact of mode of administration on responses to health-related questionnaires, most of which have been designed to be carried out using pen and paper. For example, Ryan et al. [22] compared responses from a paper and an Internet version of the SF-36 health questionnaire among adults and reported that the ‘electronic version was equivalent in performance and more effective than the paper version’ (p. 19). Raat et al. [23] found that among children, there was little difference in the scale scores of the Child Health Questionnaire Child Form (CHQ-CF) between paper and Internet administration.

Most applications of the KIDSCREEN-27 measure of HRQoL are through paper questionnaires, and the results reported in the literature pertain to this mode of administration. To date, the application of the measure embedded within an Internet survey has not been done. The main aim of this paper is to examine the psychometric properties of an Internet version of the KIDSCREEN-27 that was originally designed as a paper questionnaire.

Methods

Target population

The target population was all P7-age children in Northern Ireland in June 2008. Approximately 24,000 pupils attended 918 primary schools that had P7-age children, including hospital and special schools. In addition, the target population included children in this age group who were being taught at home, as well as those in special educational units due to exclusion from mainstream schools.

Fieldwork

The fieldwork for the survey was undertaken throughout June 2008, and two advance letters were sent out to schools in May 2008. The first letter gave information about the survey and explained that the fieldwork would be carried out in June. The number for a dedicated telephone line was included so that principals could contact the survey team directly for more information. The letter also offered principals a confidential summary report based on their school’s responses in anonymised form in an effort to encourage their participation.

The second letter was sent out 2 weeks after the first one along with an instruction document for teachers and parental consent forms. A KLT bookmark for every P7 pupil in the school was also included to publicise the survey. Each school was allocated a unique four-digit identification code number that the children used to log onto the survey. A trouble-shooting document for teachers and parents was available on the KLT website. Consent for the children to participate in KLT involved three levels. First, the school principal agreed that the school could participate and secondly, a parent or guardian of each P7 pupil within these schools signed a consent form to say that s/he agreed that the child could take part, and returned this to the school. Finally, at the start of the questionnaire, the child was asked if s/he agreed to take part. The children who participated in the KLT survey were, therefore, those who gave their explicit consent and whose school principals and parents or guardians had already given their permission. The initial fieldwork period lasted from the 2nd to the 16th June 2008. However, due to requests from schools, the fieldwork period was extended until the end of June. An email was sent out to all primary schools on the 9th June to remind them about the survey, and thanking them if they had already taken part. Ethical approval to carry out the survey was obtained from the Ethics Committee located within the School of Sociology, Social Policy and Social Work at Queen’s University Belfast.

The survey

In order to make the questionnaire visually attractive, fun and interesting to complete, the company that designed the web version used cartoon characters from ‘Bang-on-the-door’ that is familiar to most P7-age children as they appear on merchandise targeted at this age group. In addition to reading the questions on the screen, the children could also listen to the questions spoken by actors (one man and one woman). In this way, children who had sight difficulties or who had problems reading could take part in the survey. To sustain interest, the questions were presented in different ways such as faces on a scale, words that could be ticked and text boxes into which the children typed their open-ended responses.

Once the child agreed that s/he wanted to take part, s/he was asked to enter the four-digit school identification number assigned by the KLT team to each primary school in Northern Ireland. When the code was entered, the full name and address of the school appeared on the screen, and the child was able to confirm that this was the correct school. The class teacher or other staff member could phone the KLT helpline number at any stage to confirm their school’s number. However, no further identification information or names of children were sought; the survey was therefore anonymous as no individual child could be identified.

Questionnaire content

The questionnaire consisted of 77 questions and took about 20–25 min to complete. It was designed by the KLT team, and the questions focused on bullying in school, children’s experience of their years at primary school and their attitudes towards the transfer test—which is the examination taken by many children in Northern Ireland to determine whether they are eligible for a grammar school place. These topics reflected the issues identified by the children we consulted in focus groups in January 2008 as being important to them. Each question had a ‘skip’ option which the children could use if they did not want to answer it. The KIDSCREEN-27 measure was embedded within the KLT questionnaire. To ensure consistency with the paper version of the KIDSCREEN-27, the questions were asked in the same order and used the same words as the original, although one ‘Bang-on-the-door’ character appeared on the screen alongside each question to match the rest of the KLT survey (an example of the questionnaire is available at http://www.ark.ac.uk/klt/children2008/survey.html). The questions were read out by the actors, and the children clicked on their answer.

Response

Of the 918 primary schools with P7-age pupils in Northern Ireland, 217 participated in the survey, representing 25% of schools. There were approximately 24,000 P7 pupils within the schools, of which 3,461 accessed the questionnaire. Twenty-one children opted out of taking the survey, which means that 3,440 completed the questionnaire. This represents approximately 14% of all P7 children in Northern Ireland. Within the 217 responding schools, there were 8,534 P7 children; therefore, around 40% of these children participated in the survey. While the response rate was lower than had been anticipated, the spread of schools in terms of size, location and free school meals entitlement matched fairly well with figures obtained from the Department of Education in Northern Ireland. Girls were more likely than boys to participate in the survey (56 and 44%, respectively) as were children who achieved a Grade A (the highest grade) in the transfer test—full details of the method and response rates have been published elsewhere [24].

Statistical analyses

To date, no published studies have used the KIDSCREEN-27 embedded within an Internet survey and therefore the data acquired from KLT were compared with the findings reported in the KIDSCREEN handbook for the paper version of the questionnaire [5]. The KIDSCREEN-27 was analysed using the syntax files supplied on the compact disk that accompanied the KIDSCREEN handbook, and the scores that are presented in this paper are the international T-values [5]. The T-values for each of the five domains of the KIDSCREEN-27 for the KLT sample were compared with the European norm data provided on pages 149 and 153 for children aged 8–11 years in the KIDSCREEN handbook [5]. Cohen’s D was used to test whether the mean scores on each of the five domains differed significantly across the two sample populations. The psychometric properties of the KIDSCREEN-27 data collected via the Internet survey were tested using principal component analysis (PCA) and Cronbach’s alpha. PCA was used to find out whether, in line with the results of studies using the paper version of the KIDSCREEN-27, a five-factor solution could be achieved from the online survey among this population of 10- and 11-year-old children. The internal consistency of this online version of the KIDSCREEN-27 was tested using Cronbach’s alpha. By convention, scales with reliabilities of .70 or over are considered acceptable [25]. Finally, the item non-response on the Internet version of the questionnaire was compared with the figures reported for the paper version on page 47 of the KIDSCREEN handbook [5].

Results

PCA

The KIDSCREEN team [4] reported that the paper version of the KIDSCREEN-27 has five domains. Given that this is the first time the instrument has been used within an Internet survey among a population of 10- and 11-year-olds in Northern Ireland, the KLT survey data were analysed using exploratory factor analysis. To check the suitability of the data for factor analysis, the Kaiser–Meyer–Oklin (KMO) measure of sampling adequacy and the Bartlett’s test of sphericity [26] were carried out. The KMO value was .93, which exceeds the value of .6 suggested by Pallant [25]. The Bartlett’s test of sphericity was statistically significant (P < .001), supporting the factorability of the correlation matrix.

To establish the number of components to be retained for the rotation procedure, Pallant [25] suggests three methods of selection—scree test [27], Kaiser criterion (retains all factors with eigenvalues greater than 1.0) [28] and parallel analysis using the Monte Carlo PCA programme [25]. The 27 items of the KIDSCREEN were entered into SPSS version 16.

The PCA resulted in six factors with eigenvalues above 1 explaining 31.3, 7.37, 5.47, 5.31, 5.17 and 3.79% of the variance, respectively (Table 1). However, the scree test suggested a five-factor solution which was confirmed by the results from the Monte Carlo parallel analysis (Table 1), which showed that five factors had eigenvalues exceeding the corresponding criterion values for a randomly generated data matrix of the same size (27 variables and the maximum respondent number allowed by the programme—2,500) [25]. As a result of these tests, a 5-factor solution was accepted.

Table 1 Comparison of eigenvalues from PCA analysis and parallel analysis
Full size table

The components were rotated to orthogonal simple structure using the varimax technique for ease of interpretation, and the results are presented in Table 2. The variables loaded on the factors they would be expected to and as reported by the KIDSCREEN team. A few items loaded on more than one factor but only the highest loadings on each factor are presented here.

Table 2 Rotated component matrix
Full size table

Internal consistency reliability

Cronbach’s alpha was used to test the reliability of the 5 domains, and the results are presented in Table 3. Most are similar to those reported in the KIDSCREEN handbook for the 27-item paper version of the questionnaire—the only difference was the school environment domain—nevertheless, a Cronbach’s alpha value of .76 is acceptable [25].

Table 3 Comparison of reliability scores for the European norm data (based on figures produced on page 47 of the KIDSCREEN manual for the child and adolescent version of KIDSCREEN-27 [5]) and KLT
Full size table

Comparison of domain scores

Table 4 compares the results for all children aged 8–11 years in the Europe-wide sample [5] from 13 countriesFootnote 1 with those from the KLT sample. Children in the KLT sample have slightly higher scores on the social support and peers domain, and this may pertain to the ages of the children—10 and 11 years for KLT and 8–11 years for the European reference population. However, overall, the KLT scores appear to be similar enough to suggest that completing the KIDSCREEN-27 online does not adversely affect the domain scores.

Table 4 Comparison of domain scores for the European norm data (based on figures produced on page 47 of the KIDSCREEN manual for the child and adolescent version of KIDSCREEN-27 [5]) and KLT
Full size table

To test for any significant effect sizes, Cohen’s d was calculated using the domain mean and standard deviation scores for the European sample of 8- to 11-year-olds and the KLT sample. By convention, a small effect size is taken as .2, a medium effect size is taken as .5 and a large effect size as .8 [29]. The Cohen’s d effect sizes ranged from .00 (physical well-being and school environment domains) to .12 (social support and peers domain). Overall, the effect sizes for all the domains were very small, suggesting that there is little difference between the scores reported by the KIDSCREEN group for the paper version and those obtained from the Internet version.

Gender differences

Table 5 shows gender differences between the two samples on the five domains, and the gender differences noted within the European norm data are replicated in the KLT data with boys having higher scores than girls on the physical well-being and the psychological domains and girls having higher scores than boys on the autonomy and parents, the social support and peers, and on the school environment domains.

Table 5 Gender differences in domain scores for the European norm data (based on figures produced on page 47 of the KIDSCREEN manual for the child and adolescent version of KIDSCREEN-27 [5]) and KLT
Full size table

Item non-response

In terms of item non-response on the KIDSCREEN-27, this is generally good as reported in the KIDSCREEN handbook (page 47) for the paper version of the child and adolescent questionnaire which had total item non-responses ranging from 1.72 to 3.83% [5]. The item non-response is even lower for the KLT survey—1% when the survey was carried out online.

Discussion

These results suggest that the data collected using the KIDSCREEN-27 embedded within a more general Internet survey for children are comparable to those produced for the European-wide sample using paper questionnaires. In line with the findings reported by the KIDSCREEN team, the data from KLT supported a five-factor solution, and the reliability estimates were the same as, or similar to, those published in the literature [3–5].

The means for the KLT domain scores were comparable to those reported for the 13 countries in the Europe-wide sample. Although there were undoubtedly a number of differences between the two samples in relation to age, country and ethnic origin, nevertheless, the results from the tests for significant effect sizes across the two populations showed that these were low. In addition, the gender differences reported in the literature were replicated in the KLT with girls having higher mean scores than boys on three domains (autonomy/parents, social support and peers, and school environment) and boys having higher mean scores on two domains (physical health and psychological well-being). The item non-response for the Internet version of the survey was lower than for the paper version which supports the work of Raat [23] who reported similar findings when he compared an Internet with a paper version of the CHQ-CF.

Limitations and suggestions for future KLT surveys

There were some limitations to the method employed in this survey. The first is that to ensure the text was large enough for children with sight problems, the five response categories were not in a straight line as they are in the paper questionnaire. This compromises the scale somewhat, and future KLT surveys using the KIDSCREEN need to ensure that the responses are all on one line to ensure direct comparability with the paper version. A second limitation is that all the children completed the survey online, which means there was no direct comparison between the Internet and paper versions among this population of children. Therefore, we had to rely on comparisons with the European norm data collected from the paper version as published in the KIDSCREEN handbook. As we intend to run the KLT survey on an annual basis and include the KIDSCREEN in future years, there is scope to design a mixed-mode approach with some children assigned to completing a paper version of the measure and the rest doing it online. Furthermore, if future studies use the computer version of the questionnaire that has been designed by the KIDSCREEN team, then more direct comparisons with the animated version may be possible. There is also scope for carrying out confirmatory factor analysis of the KLT version of the KIDSCREEN-27 within this population of P7 children in future years.

A third limitation is that the published data were for 8- to 11-year-olds, while the children who took part in KLT were aged 10 and 11 years, which means that direct comparisons cannot be made. Finally, the response rates for schools and children in 2008 were low and do not allow us to presume that our achieved sample is representative of the population of all P7 children in Northern Ireland. The KLT team has carried out an evaluation of a sample of schools that did not take part in the survey, and the information obtained from this exercise will be used to boost response rates in future surveys.

Conclusion

Notwithstanding the limitations outlined above, the results from KLT provide a first attempt at assessing the psychometric properties of the KIDSCREEN-27 embedded within an online survey of children and suggest that this is a valid method of collecting quality of life data from them. While more research is needed using different populations of children, nevertheless it would appear from these preliminary results that surveys can be made interesting and fun for children to complete without compromising the quality of data obtained from standardised questionnaires.

There is scope for wider applications of the KLT survey both in its present format and with modifications. The survey could easily be extended to other age groups to test the correlates of health and HRQoL among children in the general population. It could also be easily adapted to identify children with particular health and well-being problems who would benefit from early intervention programmes that could lead to more successful outcomes in later life.