1 Introduction

There is growing awareness that measuring child well-being is an important basis for planning children’s services and charting their impact (Bradshaw and Mayhew 2005; Costello et al. 2006a, b; Ford 2008; Rutter and Stevenson 2008). In children’s services, most such work has been under the banner of need and outcomes assessment. Population assessments of need provide critical information for making hard decisions about resource allocation (Percy-Smith 1996; Gould 2001). They can help shape the nature of services to be provided and also provide baseline data against which to assess service effectiveness. This is reflected in the popularity of measuring need and outcomes in the context of planning children’s services in numerous western developed countries, including Australia, Canada, England and Wales, Ireland and Sweden (Axford 2010a).

In England and Wales, where local authorities are legally required to identify and assist ‘children in need’, many needs assessment studies are conducted at a local level. However, their nature and quality vary considerably; for example, a recent empirical analysis of 83 needs assessments from two contrasting local authorities found a predominance of small-scale analyses that use qualitative methods to ascertain what service-users feel they need (Axford et al. 2009). Large-scale quantitative surveys using standardised measures with representative samples to gauge the well-being of children in a discrete area relative to norm data were rare. Although more structured quantitative and qualitative needs-assessment methods have been developed (Little et al. 1999, 2002), their use is patchy (Utting 2009).

A connected problem is the gap between measuring need and outcomes and acting on the results (Axford 2010a): this may be attributed to a lack of methodological rigour of studies or weak justifications for their implementation, such as to address a budget underspend or to justify decisions already taken. The result, often, is low trust in the data and a perception that they are neither relevant nor useful. This is part of a much larger set of challenges regarding getting research into policy and practice: simply producing a report, or even a summary of a report, and sending it out into the ether in the hope that it will be seized upon is not sufficient (Bullock 2006; Chaskin and Rosenfeld 2007; Nutley et al. 2007).

This article reports an attempt to develop and implement a new instrument that would address acknowledged weaknesses in needs and outcome assessments and feed directly into a structured service development process. It outlines the instrument and the method of implementation, describes the functions of the work and discusses its strengths and weaknesses. Findings from an application of the approach in one local authority in England and the process of using the data to inform children’s services planning there are discussed elsewhere (Hobbs et al. 2010; Axford and Morpeth forthcoming).

2 Context

The main policy driver for the work was the greater focus in children’s services in the UK on outcomes defined in terms of children’s well-being. This was a reaction against services being driven by outputs—administrative or performance indicators, such as the number of children cared for away from home (Axford and Berry 2005). The catalyst was an inquiry into a major child death scandal (Laming 2003) but the shift coincided with accumulating evidence that aspects of children’s well-being in the UK had declined over the previous 25 years (Collishaw et al. 2004) and that it is lower than in many other developed countries (UNICEF 2007). In England and Wales this outcome focus is now enshrined in legislation, although official measures continue to focus largely on process (see HM Government 2008).

Multi-agency working, also encouraged by the legislation, has become more common, reflecting a recognition that the often complex needs of children and families require a coordinated response by social services, health, education and youth justice (Percy-Smith 2000; Morpeth 2004; Anning et al. 2007). Local children’s services agencies are encouraged to plan services in a coordinated way under a single dedicated director, matching services to needs and monitoring the impact on outcomes.

In addition, the emphasis on improving outcomes has prompted increased attention to evidence-based programmes (Social Exclusion Taskforce 2007).Footnote 1 In order to know where best to target these services, it is important to gather good quality data on child well-being, need and the scope of potential target populations. There is also growing recognition that such data can be used to monitor trends in well-being and so gauge the impact of policy and practice changes in terms of child outcomes (Little et al. 2005). This chimes with policy makers’ avowed interest in finding out what works, notwithstanding a somewhat flaky record in this regard—at least in the UK (Rutter 2007).

The research context for the work reported here was the emergence in recent years of several new service development methods (Renshaw 2008). Increasingly referred to as ‘prevention operating systems’, these are structured processes to help communities, agencies or local authorities improve child outcomes by developing (or selecting), implementing and then evaluating effective prevention, early intervention and treatment models (Fagan et al. 2008). Most represent attempts to connect prevention science and community engagement (Weissberg and Greenberg 1998; Spoth and Greenberg 2005), in other words to harness research evidence—in this case population needs assessment or epidemiological data—and enable innovative professionals and laypeople to use that evidence in a structured process to design new interventions, or adapt existing models, and develop a sense of ownership of the resulting services. They fit into the ‘organisational excellence’ model of research utilisation, in which leaders in the organisation shape the research and foster a research-minded culture (Walter et al. 2004). They also involve the strategy of facilitation, which includes technical, financial, organisational and even emotional support and appears to support the effective use of research evidence (Nutley et al. 2007).

The local context for the work reported in this article was the desire of children’s services leaders in one local authority to use the Common Language operating system (see Axford and Morpeth forthcoming). The local authority concerned is the largest in the European Union, with a population of over one million and approximately 230,000 children (aged 0–15). Residents are from a wide range of ethnic and religious backgrounds. Common Language requires an analysis of child well-being early in the process by way of an epidemiological survey of children’s outcomes (their health and development), some potential influences on those outcomes and, to a lesser extent, services received. Specifically, the functions of the survey data are:

  1. (1)

    To provide a ‘broad brush’ picture of the health and development of a representative sample of children aged 0–18 and the potential influences upon this;

  2. (2)

    To offer one lens through which to consider the prioritisation of improvement in specific child outcomes when it is impossible to focus on everything;

  3. (3)

    To assess how aspects of health and development and influences upon this may be related, offering potential sources of intervention;

  4. (4)

    To give a sense of the size and characteristics of potential target populations, so allowing an estimation of realistic targets in improvement of child well-being;

  5. (5)

    Over future applications of the method, to provide banks of cross-sectional data with which to monitor trends in child well-being. When these are linked to administrative or identifying data, longitudinal datasets may be generated.

In outline, the work involved:

  1. (1)

    Engagement of local policy makers, senior practitioners and parents and children;

  2. (2)

    Development of a robust measure to be completed by parents on behalf of children aged 0–6 (the Health and Development Outcomes in the Community measure—HDOC);

  3. (3)

    Development of a robust self-report instrument for children aged 7–18 for completion in schools (the Health and Development Outcomes in School measure—HDOS);

  4. (4)

    Engagement of a survey company to administer the 0–6 HDOC survey in parents’ homes, and development of a web-based application and support system for implementing the 7–18 HDOS survey on-line in schools;

  5. (5)

    Identification of samples representative of children aged 0–6 and 7–18 in the local authority;

  6. (6)

    Analysis of the results and development of a template for the local authority to generate individualised school reports for inspection purposes;

  7. (7)

    Dissemination of results in accessible formats for relevant audiences.

3 Instruments and Scales

In order to assess robustly a broad range of children’s outcomes and potential influences on those outcomes, a comprehensive, reliable and valid instrument was required. It was not feasible to develop new instruments from scratch as this would involve a large investment in item development and testing their ecological validity, practical utility and psychometric properties, and the initial study had to be completed within 4 months to fit the service planning cycle. It was therefore decided to construct an instrument that mainly comprised pre-existing standardised scales. Six criteria determined their selection.

First, in combination the scales needed to cover the five main outcomes enshrined in the Government’s Every Child Matters (ECM) outcomes framework and ensuing legislation (being healthy, staying safe, enjoying and achieving, making a positive contribution and achieving economic well-being) and, where possible, the 25 sub-outcomes.Footnote 2

Second, scales needed to display proven validity, in other words to measure what they are supposed to measure. For example, a measure of hyperactivity should identify issues related to inattention, concentration and restlessness but not necessarily anti-social or aggressive behaviour.

Third, scales had to have proven reliability in terms of consistently producing the same results, particularly when applied by different researchers or to the same subjects at different points in time (when there is no other evidence of change). Internal consistency and test-retest alpha coefficients above .70 are typically judged to be adequate. The scales also needed to be sensitive to change, so that if repeated over time they would pick up any changes in children’s well-being.

Fourth, scales had to be developmentally and culturally appropriate; as far as possible, their reliability and validity must hold for different age and ethnic groups within the target population.

Fifth, the instrument had to be practical. Component scales needed to be relatively brief in order to keep the instrument short enough to be completed by parents or children in 45 min or less. They also needed to be presented in a simple and consistent format, typically using a Likert scale. The scores or responses they elicited were also required to be appropriate for calculating mean scores and, ideally, a cut-score indicating impairments to health and development (although in practice very few measures contain the latter).

Sixth, it was desirable that scales had published (or at least available and reliable) comparative data. This would help decision-makers judge whether a score of, say, 3.5 on a particular scale is good or bad. Normative data typically comprise published mean scores and standard deviations and, where available, cut-scores indicating the proportions of children with likely impairments to health and development. UK national norms matching the age and diversity of the sample would be optimal but few scales have such data.

In order to identify suitable scales using these six criteria, the research team searched the standard online databases of academic literature (including psychINFO, Science Direct, PubMed and Web of Science). This process resulted in a shortlist of 85 separate scales. Drafts of the 0–6 HDOC and 7–18 HDOS instruments were prepared using the optimal combination of scales according to the aforementioned criteria. They were then shared for comment with primary stakeholders in the local authority, including an Assistant Director of Children’s Services and the Director of the Research and Statistics Department. Changes were incorporated where possible but without altering the wording or structure of composite scales. The final instruments are available by request from the authors.

3.1 Health and Development Outcomes in the Community (HDOC); 0–6 years Parent-Report Instrument

The constituent scales of the 0–6 years parent report HDOC are now described. Rather than report all previously published reliability and validity statistics for each scale, references are provided for studies examining these; unless otherwise stated all of them demonstrate adequate reliability and validity according to accepted standards (Cronbach alpha coefficients over .0.7 for internal validity and intra-class coefficients over 0.5 for test-retest reliability).

The Pre-school Children’s Quality of Life Questionnaire (TAPQOL; Fekkes et al. 2000; Bunge et al. 2005) measures a parent’s perceptions of their child’s quality of life across 12 primarily health-focused dimensions: (i) stomach problems; (ii) skin problems; (iii) lung problems; (iv) sleeping problems; (v) appetite; (vi) problem behaviour; (vii) positive mood; (viii) anxiety; (ix) liveliness; (x) social functioning; (xi) motor functioning; and (xii) communication. For each dimension there is a subscale comprising between three and seven items (a total of 43 items). The scale was developed for children aged between 9 months and 6 years, with the latter three subscales only suitable for those aged over 18 months. Responses are on a three-point Likert scale (Never, Occasionally, Often), resulting in a score between 0 and 100 (higher scores indicate better functioning in that domain). Comparison data are only available from a non-representative sample of children from the Netherlands (Fekkes et al. 2000).

The parent-report version of the Strengths and Difficulties Questionnaire (SDQ; Goodman 1997) measures the mental health of children aged 4–16. A slightly modified version is available for parents of children aged 3–4 (this version was used for all 4-year-olds). The SDQ contains 25 items arranged in five sub-scales covering behaviour/conduct problems, inattention-hyperactivity, emotional well-being, peer problems and pro-social behaviour. Each sub-scale contains five items to be answered across a three-point Likert scale (Not true, Somewhat true, or Certainly true), resulting in a score between 0 and 10. The first four sub-scales are based on medical diagnoses from two primary classification schemes: the DSM-IV (American Psychiatric Association 1994) and the ICD-10 (World Health Organization 1993) and may be combined to form a broad overall measure of a child’s mental health (scored 0–40, with higher scores indicating greater need).

The SDQ can be used to calculate for each sub-scale and the ‘total difficulties’ score both the average amount of difficulty and also the proportion of children displaying ‘borderline’ (some need) and ‘abnormal’ (high need) scores. Children in the ‘abnormal’ category are likely to reach the threshold of a clinical diagnosis for a mental disorder (Goodman et al. 2000a, b; Goodman 2001). The SDQ has been used widely in Europe, North America and many other countries and is translated into over 60 languages. The paper version is freely available.Footnote 3 Normative data are available for parents, teachers and children in the UK and numerous other countries. Validity and reliability data are provided in these and several other studies (Goodman 1999; Smedje et al. 1999; Muris et al. 2003; Bourdon et al. 2005).

The 10-item Parenting in Poor Environments scale (PPE; adapted from Ghate and Hazel 2002) asks parents about aspects of their local environment, including crime, vandalism, prejudice/abuse, drugs, pollution and the state of houses and buildings. Parents respond using a three-point Likert scale (No problem, Bit of a problem, Serious problem), resulting in a score between 0 and 20 (higher scores indicate a poorer environment). Although no reliability data are reported, the PPE was validated against a random stratified sample of 10,000 Enumeration Districts in the England, Scotland and Wales (at the time the smallest areas for which national census data existed). Comparative data are available from the aforementioned national study, which focused on disadvantaged areas.

The Parenting Stress Scale (PSS; Berry and Jones 1995) contains 18 items representing positive aspects of parenthood—emotional benefits, self-enrichment and personal development—as well as negative aspects—demands on resources, opportunity costs and restrictions. Respondents use a five-point Likert scale (from Strongly disagree to Strongly agree) to indicate how far they agree with each statement. This yields a score of between 18 and 90 (higher scores indicate greater parental stress). Comparison data are available from a study of parents of children aged 0–6 years in a deprived area of Dublin (Axford and Whear 2006).

The Misbehaviour Response Scale (MRS; Creighton et al. 2003) was developed from the parent-child Conflict Tactics Scale (Straus 1979; Straus et al. 1980) in order to be more applicable in a UK context. Twelve items were selected asking parents about the frequency of various responses to their child’s misconduct. Local authority stakeholders requested that questions related to severe violence be omitted due to ethical concerns. Parents use a five-point Likert scale (from Never to Happened more than 20 times in the past year) to respond to statements. The 12 items used produce three higher-order sub-scales assessing verbal reasoning, psychological aggression and physical aggression.Footnote 4 Normative data for the MRS have not been published but comparison data from corresponding items are available from a representative sample of children in Dublin (Berry 2009a).

Material deprivation was assessed using the 10-item adult perceived deprivation subscale of the Family Resources Survey, which covers areas such as ability to keep the home warm, repair broken goods and make savings (McKay and Collard 2004). All items are based on research into what the wider population considers to be essential for living in modern British society (Gordon et al. 2000). Families unable to afford two or more items were judged to be in poverty.

The HDOC includes a number of additional questions besides these standarised scales. Demographic questions were adapted from the national census, for example on ethnicity, carer status, age and socio-economic status. There are questions about the number of rooms (excluding kitchens and bathrooms) and residents in order to assess whether the child lives in an overcrowded dwelling. Overcrowding was estimated by dividing the number of rooms (excluding kitchens and bathrooms) by the number of people, with a result of greater than 1.5 indicating that children and families are likely to be living in overcrowded accommodation (ODPM 2004). Questions from the parental questionnaire of the Effective Provision of Pre-School Education study (EPPE; Sylva et al. 2004) were incorporated to assess attendance at pre-school activities or centres, whether parents read to or teach their children at home and whether special educational or learning needs have been identified. Finally, parents’ aspirations for their children were determined in relation to further education, training and voluntary work (Bosworth and Espelage 1995).

Table 1 indicates how scales and questions from the parent-report HDOC instrument for children aged 0–6 map onto the five ECM outcomes.

Table 1 Parent-report measures for children aged 0–6 against ECM outcomes
Full size table

3.2 Health and Development Outcomes for children in School (HDOS); Self-Report Instrument for Children Aged 7–18

There are two routes through the school-based self-report HDOS: children aged 7–11 are asked a narrower range of developmentally appropriate questions, while those aged 12–18 answer additional questions deemed more appropriate for this age-range (e.g. about drug and alcohol use).

The Kidscreen-52 questionnaire (Ravens-Sieberer et al. 2005; The Kidscreen Group 2006) is designed to measure the subjective health and well-being of children and adolescents across a range of domains. It is appropriate for children aged 8–18, has been standardised in 13 European countries and comprises 52 items assessing the child’s subjective perception of: physical well-being; school environment; bullying; financial resources; self-perception; psychological well-being; moods and emotions; social support and peer relations; and autonomy. Only the first four of these subscales were used in both routes through the HDOS, and the self-perception subscale was used with those aged 7–11. Responses are on a five-point Likert scale (from Not at all or Never to Extremely or Always). Each sub-scale generates a score of 0–100 (higher scores indicate better subjective well-being in that domain). Normative comparison data are available for Europe as a whole and for each of the 13 countries in the research (including the UK).

The child self-report version of the Strengths and Difficulties Questionnaire (SDQ; Goodman 1997) is for children aged 11–15. It is structured like the parent version described earlier but worded slightly differently to reflect the self-report nature. It demonstrates adequate published reliability and validity with the exception of peer relations (.41), and ICCs with parent report range from .30 to .48 (Goodman 2001). The self-report version has also been used cautiously with children aged 7–11 (Mellor 2004; Muris et al. 2004). The scale developer granted a licence to use the online version of the measure, which is accompanied by pictures to aid understanding (Truman et al. 2003).

The Things I’ve Seen and Heard scale (TISH; Richters and Martinez 1992) was designed to measure the frequency of exposure to violence and crime at home and in the community and has been used with children aged 6–8 and 11–17 (Richters and Martinez 1992; Berry 2009a). It provides two scores: one to indicate the frequency of children’s exposure to violence in the home and community, and one to indicate perceived safety. Children respond across a four-point Likert scale (Never to Many times/all the time), producing a score of 0–36 on the exposure sub-scale and 0–12 on the safety sub-scale (higher scores are positive, indicating lower exposure and greater safety). The HDOS incorporated nine items from the original 16-item exposure scale (items relating to death, guns and stabbing were excluded due to ethical concerns expressed by local authority stakeholders). These nine items were completed by children aged 12–18 only. The safety scale comprises four items and was completed by all children. The TISH has mostly been used in the US where limited comparison data are available (Richters and Martinez 1992). For comparative purposes the present research used data from a study that applied the TISH with a representative sample of children aged 11 to 17 in Dublin (Berry 2009a).

The Revised Personal Lifestyle Questionnaire (PLQ; Mahon et al. 2002a, b) measures positive health practices in young adolescents in domains such as exercise, nutrition and substance use. It was developed in the US for use with young people aged 12–14 but the language used makes it cautiously appropriate for those aged 7–18. The HDOS employs the nutrition sub-scale only, which contains four items answered using a four-point Likert scale (Never to Always) and yields a score of 0–12 (higher scores reflect better diet). The internal consistency coefficient for the nutrition sub-scale is poor at .37 while the overall one-factor construct validity of the scale ranges between .38 and .54 (Mahon et al. 2002b). These modest reliability and validity scores reflect the difficulties of succinctly measuring nutritional behaviours in children (Nicklas 2004). Comparison data are available from a non-representative sample of children aged 12–14 from the US (Mahon et al. 2002a).

The Parenting Style Inventory (PSI; Darling and Toyokawa 1997) measures children’s perceptions of the parenting styles they experience. Parenting style refers to ‘a constellation of attitudes toward the child that are communicated to the child and that, taken together, create an emotional climate in which the parents’ behaviours are expressed’ (Darling and Steinberg 1993, p. 488). The PSI was designed for use with children over 11 years and has been used with school-based populations of US children. It comprises 15 items arranged in three sub-scales: demandingness (relating to boundary setting, praise and punishment); responsiveness (relating to interactions with the child); and autonomy-granting (relating to child’s scope for independence). Children respond using a five-point Likert scale (ranging from Strongly disagree to Strongly agree).

Higher scores on each subscale indicate a greater level of each style of parenting. This scale is only used only in the 12–18 years route of the HDOS. Comparison data are available from a rural/suburban population of children aged 11–14 in the US (Darling and Toyokawa 1997).

The school-age version of the Personal Well-Being Index (PWI; Cummins and Lau 2004) provides a global measure of the subjective well-being of children aged 5–18. It comprises eight items and covers seven domains: standard of living; personal health; achievement in life; personal relationships; personal safety; feeling part of community; and future safety. Responses are arranged on an 11-point Likert scale (from 0 to 10), producing a global well-being score of 0–100 (higher scores indicate more positive subjective well-being). The scale is used for all children aged 7–18 in the HDOS. The scale developers provided unpublished comparison data from Australia.

In addition to these standardised scales, the HDOS instrument includes demographic questions that cover age, gender, school-based data and overcrowding. Children are also asked whether they have had an accident in the last 12 months requiring a visit to hospital, if they have been involved with the police and what their future aspirations are as regards further education, training and voluntary work (Bosworth and Espelage 1995). Those aged 12–18 are asked about frequency of cigarette smoking, alcohol consumption and illicit drug-use.

Table 2 indicates how scales in the school-based HDOS instrument for children aged 7–18 map onto the five broad ECM outcomes.

Table 2 Child self-report measures for children aged 7–18 against ECM outcomes
Full size table

Child well-being data from the HDOS survey were linked, via unique child identifiers, to administrative data held by the local authority, including academic performance, eligibility for free school meals, special educational need status, looked-after status and ethnicity.

4 Sampling

4.1 Parents of Children Aged 0–6 in the Community

The survey required parents/carers from a representative sample of children aged 0–6 in the local authority, ensuring good coverage of different parts of the city, socio-economic status, ethnic groups and family type. Quota sampling was used in order to ensure representation of children and families across the 40 wards of the city, resulting in sample sizes in the region of 500 in each year of application (subsequent years also defined quotas according to age and ethnicity). This method of sampling is a form of stratified sampling where relative proportions of people are sampled from the population based on a set of specified criteria / quotas. Although it is a non-probability sampling method (and a random sample method would be more robust) it was deemed fit for purpose, particularly given cost and time restraints.

4.2 Children Aged 7–18 in School

The school-based sample was intended to include a representative selection of local authority maintained and independent schools, and the majority of children aged 7–18 within the schools selected. There needed to be good coverage of different parts of the city, levels of deprivation (as measured by proportion of children receiving free school meals), ethnic groups, school type (primary/secondary, local authority/independent) and school performance level (assessed by the proportion of children reaching expected levels in national key stage assessments). Schools were stratified across these criteria and a random 10% were selected to form a list of schools to be approached.

Schools in the sample were requested to ask all children in Years 4–6 of primary school (ages 7–11) and Years 7–13 of secondary school (ages 11–18) to complete the HDOS instrument on-line during lesson time. Response rates of 70% or more were required within each school (and across year groups) in order to produce a broadly representative sample. Schools unable to meet this threshold in the allotted time period (see below) were given additional time in order to increase response rates. Schools in which the final response rate was below 70% were excluded from the analysis. This methodology resulted in a sample of approximately 5,000 children from 25 schools and is broadly representative of the local authority as a whole. It was intended to retain this ‘core sample’ of schools in subsequent years of application. Further, the HDOS instrument was made available to other schools in the local authority for their own self-assessment purposes and to foster engagement with the local authority’s children’s strategy; this substantially increased the number of children and schools participating in the endeavour (Hobbs 2009).

5 Implementation

5.1 HDOC Instrument for Parents of Children Aged 0–6

Trained researchers undertook the survey of parents of children aged 0–6. Owing to the sensitive nature of some items in the instrument, questionnaires were left with respondents to self-complete (unless parents/carers required assistance). Completed questionnaires were collected 30–40 min later or at an alternative agreed time. In households where more than one child aged 0–6 was resident, parents/carers were asked to focus on the child whose birthday was most recent.

All data were inputted into a secure SPSS database and 10% were double-checked for accuracy of input. If there were more than 0.5% errors per questionnaire—equivalent to more than one error per questionnaire—then further checks were to be made, although in practice this was not necessary.

5.2 HDOS Instrument for Children Aged 7–18 in School

The HDOS instrument was converted into an audio computer assisted personal interview (ACAPI) format for children to complete on-line in school. Questions were presented clearly on the computer screen with a number of possible responses for the child to click on. This has several benefits compared with paper questionnaires: it is generally more popular with children; the format of questions is clearer; the audio option helps children with reading difficulties to understand the questions; children are routed automatically through the correct options according to their age and answers; and the data are collated efficiently and accurately in an electronic database (Fricker and Schonlau 2002).

Schools received an information pack prior to the survey informing teachers about the rationale for the survey, the practical requirements for undertaking it and the main benefits for the school. The survey enables the production of an anonymous individualised school report comparing the outcomes of children in that school with the local authority as a whole and, where possible, with national averages. Critically, this report contains data that UK schools are now required by Government to collect routinely for self-assessment and inspection purposes.Footnote 5 The information pack also contained a test-link for teachers to view the instrument in advance of implementation.

Children and young people completed the HDOS during lesson-time in school—typically in ICT (Information and Communication Technology) or PSHE (Personal, Social and Health Education) lessons. It took classes approximately 30–45 min for all children to complete the instrument under supervision by a teacher or learning support worker. Schools needed to have computers with a broadband internet connection and if children required the audio option the school needed to provide headphones. A paper version of the instrument was available for schools preferring this option.

The HDOS online instrument also contained a password-protected local authority administration portal. This allowed staff from the local authority Research and Statistics department to view real-time response rates and follow up with schools that had not reached the required response rate.

All data were automatically stored to a secure server for later extraction and analysis. Data on individual children were linked to corresponding local authority administrative data with an 85% success rate. The main reason for an unsuccessful match was an incorrect or false name.

Schools had 3 weeks to consider the materials and obtain forms from parents not consenting to their child taking part. They were then given half a school term (6 weeks) for data collection, with a 3-week extension if the 70% response rate threshold was not reached.

6 Ethics

The research design was scrutinised and approved by an independent ethics committee, which examined arrangements regarding child protection, informed consent, confidentiality and data protection. Particular attention was paid to balancing a responsibility to address serious concerns about a child’s welfare that may arise during data collection with protecting the confidentiality of respondents’ data (Berry 2009b).

The following processes operated in the parent-report HDOC instrument for children aged 0–6 because four questions about physical punishment from the MRS scale were deemed sufficient to potentially trigger safeguarding concerns.Footnote 6 (There were no safeguarding issues with the child self-report HDOS school instrument as no questions, alone or in combination, were deemed sufficient to trigger serious child welfare concerns.) First, at the beginning of the interview, respondents sign a statement agreeing that ‘If I indicate that my child is in serious danger I may be advised afterwards to contact someone who can help’. The assumption is that they can then choose to reveal information in full knowledge of the likely consequences. Some will choose to use this option as a call for help. Others will not disclose certain information. Second, if parent responses to any of the four questions regarding physical punishment reached a high pre-designated threshold—agreed with the lead safeguarding officer in the local authority—then the research team considered the information in relation to other evidence in the completed questionnaire on the child’s health and development. This informed a decision about whether to write to the respondent and social services.

The rationale is that this process: protects the family’s confidentiality insofar as no specific information is revealed; upholds the researcher’s ethical responsibility to take action to protect a child in serious danger; places a high degree of control with the parent/carer; allows practitioners to check to see if other people have raised concerns about the child or if other agencies are already working with the child and his or her family, so avoiding hasty intervention; and highlights for agencies that they may wish to pay special attention to the child and his or her family.

In relation to informed consent, all parents/carers approached in the HDOC survey were informed in writing and verbally (by the field workers) about the nature and purpose of the study and how data would be used (anonymously and in aggregate form, but also potentially if the aforementioned child protection concerns were raised) and given the option to consent to take part or not. In the school-based HDOS instrument for children aged 7–18 the school sent parents/carers a letter informing them about the nature and purpose of the study and the potential use of their child’s data (matched with existing local authority data and used only in aggregate form for the planning of service provision). Parents/carers were then required to return a slip to the school indicating if they did not want their child taking part. If parents gave passive consent, the children were presented with the same information upon logging onto the online HDOS instrument and given the option to consent to take part or not. Children also had the option to skip questions that they did not want to answer and to withdraw at any time, resulting in the deletion of all data provided by that child during or after completion of the survey.

7 Analysis

SPSS computer syntax was written for each stage of the analysis, enabling rapid analysis of the data. All data were then recoded and scored, taking account of missing data (only cases with all items completed within the relevant subscale were included in the analysis). Modest imbalances between the sample and the demographics of the local authority were accounted for by statistical weighting of data prior to analysis (see Hobbs 2010; Hobbs et al. 2010). This resulted in the production of mean scores on standardised measures and, where possible, estimation of the proportion of children with likely impairments to their health and development (as indicated by scores above or below specific thresholds—the tail of the distribution). Key findings were compared with available comparison data; an example of these results is presented in a related article (Hobbs et al. 2010).

8 Dissemination

A comprehensive dissemination strategy was developed for presenting the findings to a range of audiences in the local authority. There are approximately 500 variables (individual items, mean and tail scores) across both instruments, so the main messages needed to be presented clearly and manageably for the data to be used in strategy development and service design. Rather than produce an unwieldy report that few people would read, headline findings were presented visually using versatile computer presentation software (Macintosh Keynote). Findings were organised by outcome (the ECM outcomes) and influences upon outcome. For each outcome, data on the mean and the tail were compared with available comparison data and this was supplemented with salient examples from individual instrument items (hypothetical example presented in black/white in Fig. 1). The presentation was structured around a coherent narrative on the well-being of children in the local authority.

Fig. 1
figure 1

Example of how data were presented

Full size image

In addition, the research team created an online video to be viewed by all new children’s services staff in the local authority as part of their induction. This discussed key concepts regarding the data (e.g. focus on outcomes, the relationship between the mean and the tail) along with key findings and how they informed and relate to the emerging children’s strategy in the local authority.

Finally, all participating schools that reached a 70% response rate received a visually attractive individualised school report in slideshow format. This presented key outcome data at the school level and offered comparisons with the local authority and, where possible, national data. It was stressed that these reports were for the schools’ use only and not for use by the local authority to rank or compare schools. Schools received their reports within half a term of data collection finishing and could use the reports for their own self-assessment and external inspections.

9 Discussion

This article describes the development and implementation of a method to measure the well-being of children and their families as part of a structured strategy development and service design process. Its main features are summarised in Table 3. This final section considers challenges in work of this kind and describes what was done in the present study to address them and what remains to be done. It is worth noting in this context that this article focuses on the first large-scale application of the method. The same (first) version of the method has now been applied 4 years successively in the local authority and once in Northern Ireland. It was also adapted for use in Atlanta, US. These subsequent applications and updated versions of the HDOC/HDOS will be reported on in due course.

Table 3 Features of the approach
Full size table

9.1 Conceptualising Well-Being

The first challenge for large surveys of this kind is to develop a meaningful conceptualisation of well-being. The method described in this article takes into account different areas of children’s lives, objective and subjective perspectives, and can be analysed in terms of concepts such as need, poverty, quality of life and social exclusion (Axford 2008). It distinguishes between children’s health and development and influences on that health and development, but also between different areas of children’s lives—life-satisfaction, self-perception, family, home, health, friends, school and so on. Thus, although the method was developed with the Every Child Matters framework in mind, it is not constrained by time and geography.

9.2 Maximising Participation

The second challenge is to maximise participation in surveys of this kind. Poor response rates in school surveys could have several inter-related sources: a perception on the part of teachers that there is insufficient time to complete the survey; practical constraints in schools making it difficult logistically (e.g. availability of ICT facilities, the timing of national exams, movement between lessons, competing curricular requirements); concerns about confidentiality, data protection and the safeguarding children; the view that questions are too intrusive; not realising the value or potential of results (at school and local authority levels); and children having difficulty understanding and therefore answering questions, leading to missing data. If realised, any of these risks would limit the representativeness of the sample, meaning that findings may be treated with suspicion or not used at all.

In the work reported here, certain post-hoc steps were taken to enhance confidence in the findings, such as only including in the analysis schools where the response rate meets a set threshold (70%) and weighting data statistically at the local authority level to ensure representativeness. However, it was deemed preferable to be proactive and take steps at different levels from the outset. Thus, it was important to have a strong policy lead locally from the Director of Children’s Services so that schools gave the survey priority. Next, it was necessary to provide practical means of supporting survey implementation by students and teachers. The aim was to design a user-friendly, reliable and visually attractive system. Potential ICT complications include system crashes caused by faults in the design of the on-line tool, insufficient server capacity to deal with a high volume of concurrent users or data being lost due to schools having inadequate technology specifications. The agreed solution was an ICT system that requires minimal support. For example, the server was known to be capable of supporting numbers of users far higher than estimations of concurrent use. The on-line questionnaire was designed to the minimum specification necessary to be supported in schools (based on an assessment of what ICT equipment schools had). Password procedures enabled children to log back on if they did not complete the questionnaire (due to time or technical difficulties) and so continue where they left off. These features were supplemented with a teacher information pack and email / phone support from the research team and the local authority Research and Statistics Department.

Further, it was necessary to understand the practicalities of ‘real life’ in schools in order to fit within given constraints—curricular requirements, lesson times, avoiding particularly busy times of year, and so on. Ensuring that the questionnaire could be completed in less than 45 min was vital for fitting into school timetables. An option for paper versions was available as back-up for schools with poor ICT facilities, although this has significant implications for cost—owing to data processing requirements—and loses other aforementioned benefits of the on-line version. The test version preview for teachers and other local authority staff helped to identify and iron-out difficulties with the survey wording and structure.

Another major concern of teachers and other stakeholders that potentially limits participation in work of this kind is the possibility of sensitive data about children and families becoming available beyond the project team. This may occur through illicit access to the server where data are stored or through the transfer of data between the research team and the local authority. To mitigate these risks, server security in the study reported here was of the highest standard, with both internal and external firewalls, and data were extracted regularly and stored securely. Strict protocols were applied in relation to the research team and local authority sending or accessing data. All personal data were linked with unique random identifiers to maintain anonymity, with child names and postcodes stored separately and securely. All personal information was destroyed once it had served the purpose of enabling matching with administrative data. Confidentiality and data protection protocols were applied to ensure that individual children could not be identified from any reports or by ‘hacking’ into the database. Care was also taken to avoid harm or distress to children caused by participating in the survey, for example by excluding more intrusive questions. A disclosure procedure in the 0–6 HDOC survey, which contained more sensitive questions, ensured that indications of a child being in serious danger were acted upon appropriately.

Notwithstanding these efforts, other steps are needed to further promote participation in the survey. For instance, meetings and written materials could help to develop clear understanding amongst school leaders of how results will be useful in formal inspections and self-evaluation. The development of high quality, engaging materials for local authorities and schools—including materials for lessons on citizenship and social-emotional learning—would also help to encourage buy-in. Schools and local authorities would like to be able to analyse the data further (by gender, age etc.) and to do this quickly and easily on-line. Work is in progress to develop the methodology in these regards.

As regards enhancing children’s participation in the school-based 7–18 HDOS survey, there is a pressing need to ensure proper access for children with disabilities and special educational needs. Key issues are: informed consent, that is, ensuring that children understand what is being asked by doing survey, and why; ensuring that children have the cognitive skills to fully understand questions, some of which require considerable abstraction; minimising acquiescence, in other words responding randomly or to please; using audio-visuals to aid understanding; and balancing the re-wording of standardised questions without undermining scale reliability or validity. In the HDOS, for instance, the Kidscreen-52 scale has been adapted—in collaboration with the instrument designers, educational psychologists and communication experts—to incorporate visual ‘widget’ symbols and so make it more suitable for children with special educational needs (Hobbs and Jodrell 2010).

9.3 Exploiting the Data

The third challenge concerns ensuring that the data gathered are exploited as fully as possible in order to inform policy and practice. The research team sought to draw on best practice in this respect, both in terms of the presentation and distribution of the research but also the methods to encourage its use (Bullock et al. 1998; Nutley et al. 2007). It is necessary, for instance, to tailor the presentation of results to meet the needs of educationalists and other service providers. In the study reported here, survey data were presented in different formats for different audiences, and using high quality graphics in addition to text helped to communicate findings succinctly and in a way that brought them to life—for example, by showing comparisons with national and international norms, breaking the data down by age, and so on.

More significant, however, was the strong link established with local authority planning procedures. Specifically, the local authority in question used the Common Language operating system to develop its strategy for children’s services. An epidemiological study using the methods described in this article is the first step in that process. Decisions made by senior policy makers, based in part on the data, have resulted in a £40 million investment over 10 years in a series of evidence-based prevention and early programmes, with savings from reduced heavy-end services and other costs to society projected to be in the region of £100 million.

However, the data can be exploited more fully. Thus, there is the potential to link the data gathered on each child to area-based indices of well-being applied at the Lower Layer Super Output Area (LSOA) level, so making it possible to examine a range of important research and policy questions (Bradshaw et al. 2009). For example: Do children from deprived neighbourhoods or overcrowded/poor housing experience higher or lower levels of subjective well-being, emotional distress, satisfactory relationships with friends, teachers and parents, bullying and so on? And how does this vary by age, gender, ethnicity, school and local authority? Answers to questions like these will contribute to the general understanding of the impact of neighbourhoods on child well-being. (This is dependent on having gathered data on name, date of birth and postcode in order to link with local authority administrative data.)

There is also scope for analysing the match between need and services, although this requires more robust data on service use. Indeed, much more attention needs to be given to the definition and measurement of ‘service’ in children’s services broadly, for example breaking it down in terms of nature of contact, type of service received, when, where, over what period, delivered by whom, and so on. This helps to avoid the use of over-generalised categories such as ‘family support’ or ‘foster care’ (Axford 2010b).

Much of this proposed analysis depends, in turn, on tightening-up aspects of how the HDOC and HDOS instruments are used. For example, although the best available comparison data are used, these are not always optimal; they form a relative, not absolute, benchmark for comparison. Key differences in population composition (e.g. socio-economic, cultural, ethnic) and methods of data collection (e.g. timing and administration) may account for large differences between samples and points of comparison. This must be considered when reading these data. However, over multiple applications of the method banks of more site-specific cross-sectional data may be built up, so forming stronger points of comparison.

All of these tasks are necessary moving forwards but, as ever, a balance will need to be struck between scientific rigour—entailing significant expense and time—and practical utility—related to the functions of the data.