Early childhood home visiting programs aim to facilitate positive outcomes for families with infants and young children through direct support to the family, education, and referrals to other services (Supplee & Duggan, 2019). Broad goals include the improvement of maternal and child health, prevention of child abuse and neglect, and promotion of positive parenting practices, child development, and school readiness (Health Resources and Services Administration, n.d.). Home visiting programs offer support to a wide range of families, including those impacted by partner violence, substance misuse, poor mental health, and stressors related to poverty, as well as immigrant families and children with special needs (Duffee et al., 2017; Duggan et al., 2018). Although home visitors can provide support to all families with young children, many home visitors are faced with families’ complex mental health and social needs (Duffee et al., 2017). Research has shown that families affected by more complex needs benefit from home visiting programs through improved parent well-being and child development (Simon & Brooks, 2017; Whitaker et al., 2006).

However, home visitors often feel unprepared to work with families with complex needs (Korfmacher et al., 2012; Nathans et al., 2019; Tandon et al., 2005) and require ongoing support in reflecting and coping with this challenging work (Parlakian, 2001). Home visitors benefit from being able to openly communicate with their supervisors about their emotions regarding families’ experiences of trauma and stress (Watson et al., 2016); the absence of this type of processing can lead to increased stress and withdrawal (Begic et al., 2019). Yet, research findings on home visitors’ experience of professional support have been mixed. Some studies found that home visitors feel unsupported in their work and lack the necessary supervision to do their job well (Harden et al., 2010). Other research has found high levels of home visitor satisfaction with supervision overall but noted that the structure could be improved and emphasized the importance of high-quality supervision, especially for complex cases (Gill et al., 2007; Nathans et al., 2019).

The quality of home visitors’ practices and their interactions with parents and children can influence family outcomes, such as parent engagement in home visits and parenting practices. A 2004 meta-analysis found significant positive effects of home visiting on child development, parenting attitudes and behaviors, maternal education, and reducing potential child abuse, but the specific home visitor practices associated with these outcomes were not studied (Sweet & Applebaum, 2004). Research since then has found that home visitors’ efforts to develop positive relationships with families, including home visitor sensitivity, acceptance of the family, and honoring family strengths and culture, result in higher levels of parent engagement and improved parenting practices and child outcomes (Heaman et al., 2007; Kelly et al., 2008; Korfmacher et al., 2007; Roggman et al., 2016; Zajicek-Farber, 2010). When home visitors encourage parent–child interactions and align their own goals with those of the parents, families tend to be more engaged in the visits (Burrell et al., 2018; Peterson et al., 2018), which increases the likelihood of sustained enrollment in the home visiting program (Brand & Jungmann, 2014).

The alliance or relationship between the home visitor and parent is essential to home visiting effectiveness (Gomby et al., 1993; Korfmacher et al., 2007; Riley et al., 2008: Saïas et al., 2016). However, given the interpersonal complexity involved in such a relationship, it has been difficult to measure (Manz & Ventresco, 2019). The Design Options for Home Visiting Evaluation (DOHVE), which set research standards for observational instruments used in home visiting evaluations, determined that the Home Visit Rating Scales (HOVRS; Roggman et al., 2008) was one of three tools with adequate reliability and validity (James Bell Associates, 2012). Used in several home visiting evaluation and monitoring studies (e.g., Korfmacher et al., 2019; Roggman et al., 2016; Vogel et al., 2015), the HOVRS has been extensively validated to assess home visitor strategies and the relational quality between families and home visitors (James Bell Associates, 2012; Manz & Ventresco, 2019; Roggman et al., 2019). Furthermore, it is the only tool approved by DOHVE that was designed specifically to assess the quality of the relationship between the home visitor and the family (Manz & Ventresco, 2019).

Although home visitor–family relationships are key to positive outcomes, home visitors may struggle to develop relationships with families who face challenges that the home visitor does not feel equipped to handle. In fact, home visiting program staff have identified several training needs in topics related to mental health. For example, Korfmacher et al. (2012) found that almost half (46%) of home visitors wanted training in working with caregivers with depression, substance abuse, and/or domestic violence. In addition, 44% wanted training in working with families whose children have serious behavioral or mental health concerns.

These challenges are some of the topics that Infant and Early Childhood Mental Health Consultation (IECMHC) addresses. IECMHC is a relationship-based support that can enhance supervision and offer home visitors space to reflect on their emotions. IECMHC was developed to help cultivate the capacity of home visitors to better support families with varying needs, thus assisting families in fostering positive development in their children (Education Development Center, 2020; Goodson et al., 2013). IECMHC has emerged as a relatively recent strategy for addressing the socio-emotional health of children in early childhood care and education settings, and evidence is accumulating that it can be an effective strategy in other settings (for reviews, see Albritton et al., 2019; Duran et al., 2009; Hepburn et al., 2013; Perry et al., 2010).

Although the implementation of IECMHC in home visiting programs has been well-documented (e.g., Boris et al., 2006; Center for Prevention Research & Development, 2011; Education Development Center, 2020; Goodson et al., 2013; Lambarth & Green, 2019), there is a dearth of research examining IECMHC’s impact on home visiting. Findings from a recent pilot study evaluating IECMHC in a Healthy Families America home visiting program found that IECMHC significantly increased home visitors’ knowledge of mental health and their confidence in engaging parents (Lambarth & Green, 2019). In two evaluations of IECMHC in home visiting programs, most home visitors reported greater knowledge of children’s social and emotional development and child behavior in context (Center for Prevention Research & Development, 2011; Goodson et al., 2013). However, both studies collected only retrospective data on home visitors, assessing whether they gained knowledge in areas such as child development, socioemotional health, and options for behavioral health referrals (Goodson et al., 2013). More rigorous research on the effects of IECMHC in home visiting programs is needed.

The current pilot study of the Illinois Model of IECMHC was part of a comprehensive, coordinated, statewide initiative by a public–private coalition in Illinois supporting children’s mental health services to expand IECMHC across multiple systems and settings. The Illinois Model was designed to develop the skills of early childhood professionals in a range of early childhood programs, including home visiting. Its novelty is its emphasis on relationship-building, reflective practice, and program-focused consultation as a means to build staff skills (Spielberger et al., 2022). According to its theory of change, improvements in staff reflective capacity, relationships with coworkers, and knowledge of children’s and parents’ social and emotional health will lead to more positive engagement with families and children. In turn, families will have easier access to high-quality services and better outcomes.

Consultants in the pilot were master’s level professionals who provided regular consultation, training, and referrals to program staff over a period of 15 months, followed by a 6-month sustainability period of less frequent contact.Footnote 1 The Illinois Model was designed to be flexible, so the activities the consultant engaged in with program staff were determined by individual program and staff needs. Activities with home visiting programs included educational activities during staff meetings, individual-level consultation with staff about specific families, reflective consultation with program supervisors/directors, and joining staff and supervisors in their supervision sessions.

Although the pilot study included sites in both center-based early childhood programs and home visiting programs, we focus here on the components of the evaluation that were particular to home visiting programs. The primary research question pertaining to home visiting programs was the following: Does the Illinois Model of IECMHC have a positive effect on home visit quality? We hypothesized that home visitors at programs in the intervention group would have higher quality visits operationalized by more optimal home visitor practice, more positive relationships with families, and stronger family engagement.

Methods

Study Design and Participants

The Illinois Model of IECMHC was piloted in four communities that have early childhood programs in multiple systems and represent the demographic and geographic diversity of Illinois communities. The four community areas selected for the pilot were a large city, a medium-sized city, a group of rural towns, and a group of suburbs. The home visiting programs in the study were represented by Early Head Start programs and home visiting programs funded by the state board of education. We matched programs by program type(s), funding, and general demographics of children served (i.e., a large proportion of Spanish-speaking families) and randomly assigned them to either the intervention or the comparison group. Although it was not possible to achieve equal representation of each program type across all four communities, we were able to reach the desired proportion of programs assigned to each group (intervention vs. comparison). The final sample included four home visiting programs in the intervention group and two home visiting programs in the comparison group. The small number of home visiting programs in the study was due to a limited number of home visiting programs eligible for the study within the four communities, as several home visiting programs in these communities had already received the Illinois model of IECMHC.

Analytic Sample

After the program assignment, we received staff lists from the programs and invited all frontline staff and supervisors to participate in surveys and other research activities. Surveys were administered to frontline staff and supervisors at four points in time. In addition, a subsample of two home visitors (full-time staff with full caseloads) from each home visiting program was asked to participate in home-visit observations. This subsample was asked to video record two home visits, each with a different family at each of the four observation time points. Different families participated at each observation because we wanted to observe home visitors establishing relationships with families, rather than observing growth in home visitors’ relationships with the same families over time. In addition, attrition tends to be high in home visiting programs—42% (Janczewski et al., 2019) to 63% (Daro et al., 2003) within one year—and we collected data over 21 months. Participation in all evaluation activities was voluntary. A member of the research team obtained informed consent from each home visitor and each parent participating in the video recording.

Data and Measures

The data for this study were drawn from observations and surveys.

Observational Data

Home visitors video recorded two home visits at each of the four-time points, each with a different family, resulting in up to eight video recordings for each home visitor. The four-time points were baseline, Time 2 (6–8 months post-baseline), Time 3 (12–15 months post-baseline), and Time 4 (19–21 months post-baseline).

The research team used the Home Visit Rating Scales Adapted & Extended to Excellence (HOVRS-A+ v2.1; Roggman et al., 2010) to code the videos and assess home visitors’ practices and relationships with parents. The HOVRS-A+ is a widely used observation tool to assess home visitors’ strategies and relationships during home visits, such as the home visitor’s effectiveness in engaging the caregiver and child in the visit. This instrument has been validated in various program models with different cultures, including Spanish-speaking Latino families, White families in rural communities, and Black families in urban communities (Schodt et al., 2015). Internal consistency reliability has been high (Roggman et al., 2010; Schodt et al., 2015).

Two pairs of coders—one pair to code home visits conducted in English and one pair for visits conducted in Spanish—were trained in the HOVRS-A+ v.2.1 rating tool by one of the scale developers. All coders were masked to the condition of the home visitors. In this study, eight videos were double-coded for reliability, resulting in 384 scores. Of these, 378 scores were within 1 point/indicator, resulting in an inter-rater reliability with a kappa of 0.98. The coders met to discuss their ratings and any discrepancies throughout the process.

The HOVRS-A+ consists of the following seven scales with scores ranging from 1 to 7, with higher ratings reflecting higher quality practices: home visitor responsiveness to family (Cronbach’s alpha = 0.75), home visitor–family relationship (Cronbach’s alpha = 0.89), home visitor facilitation of parent–child interaction (Cronbach’s alpha = 0.82), home visitor non-intrusiveness/collaboration with family (Cronbach’s alpha = 0.87), parent–child interaction during home visit (Cronbach’s alpha = 0.92), parent engagement during home visit (Cronbach’s alpha = 0.83), and child engagement during home visit (Cronbach’s alpha = 0.91).

The HOVRS-A+ was not designed specifically to measure the effects of IECMHC. Thus, we created another scale, the IECMHC scale, comprised of the HOVRS-A+ items that we expected to be impacted by the intervention, according to the Illinois Model’s theory of change and the extant literature on mental health consultation in home visiting (e.g., Duran et al., 2009; Goodson et al., 2013; Lambarth & Green, 2019). We designed this scale to measure the extent to which home visitors engage in behaviors that promote parenting self-efficacy, encourage positive parenting behavior, and facilitate responsive parent–child interactions and the extent to which the parents in these visits engaged in more positive interactions with their children. This scale contained 13 scores: 12 individual HOVRS-A+ items and one overall domain score.Footnote 2 Cronbach’s alpha for this scale was 0.92.

Staff Survey

We administered online surveys to home visitors at all four data collection time points. Staff were sent an email with a link to the survey, followed by, at minimum, five email reminders. Survey data were collected via REDCap (Harris et al., 2009, 2019). The surveys contained nonstandardized questions about demographics, staff’s access to mental health consultation, the quality and frequency of consultation received, and their supervision. The survey also included standardized measures of burnout, depression, reflective capacity, self-efficacy, supervision, and working with children/families with challenging behaviors, which we describe in a separate paper (Spielberger et al., 2022).

Analytic Approach

We used a clustered study design with three levels: families (level 1) were nested within home visitors (level 2), and home visitors were nested within programs (level 3). We calculated Linear mixed models (LMM) for each of the outcome variables following the methodology proposed by West et al. (2007), using R version 3.6.0, nlme package, and Residual Maximum Likelihood estimation (REML). This methodology involves the following steps: (1) fit the initial unconditional (variance components) model and decide whether to omit the random home visitor effects, (2) build the level 1 model by adding parent-level covariates, (3) build the level 2 model by adding home visitor covariates, and (4) build the level 3 model by adding program-level covariates. Each of the HOVRS-A+ scales was included as outcome variables. Program-level characteristics (program size,Footnote 3 fidelity scores,Footnote 4 and treatment condition), home visitor characteristics (dosage,Footnote 5 years of experience, gender, age, race, educational attainment, and standardized measures of burnout, depression, reflective capacity, self-efficacy, supervision, and working with children/families with challenging behaviors), and family-level characteristics (age, race, education, number of children, number of hours enrolled in the program, and language)—all drawn from survey data—were included as covariates in each of the models.

To analyze the IECMHC scale developed for this study, we used SPSS, version 26.0. We included the IECMHC scale as an outcome variable in a nested (or hierarchical) two-way ANOVA to test for differences in means on the scale between the intervention and comparison groups over time, with staff nested in group to account for stronger correlations within the visits for each home visitor. This nested analysis compared differences in the scores of the visits between home visitors at programs receiving the intervention and programs not receiving the intervention. We used a multilevel analysis to account for the fact that each home visitor’s visits will correlate to their own visit scores more than they will correlate to each other’s scores. We tested for homogeneity of variance, and the assumption was met. No covariates were added, as we found no differences for any of the demographic variables on this scale score at baseline.

Results

Sample Characteristics

We received a total of 51 (27 English, 24 Spanish) video recordings of home visits; however, nine video recordings were unusable because of quality issues, short duration (we established a minimum video length of 10 min), or the home visitor or parent was not in view. Thus, we analyzed 42 videos of home visits for this study, 15 in the intervention group and 27 in the comparison group. Just over half of the home visits (52%) were conducted in English (n = 22; 60% of intervention visits, 48% of comparison visits) and 48% of home visits in Spanish (n = 20; 40% of intervention visits, 52% of comparison visits).

Table 1 displays the demographics of the sample of home visitors and parent participants from the 42 home visits. Eighty-six percent of parents had a home visitor who matched their race/ethnicity. Children in the home visits (n = 42) ranged in age from 3 to 50 months, with a mean age of 19.4 (SD = 13.4) months. About 45% (n = 19) of the sample included children under 12 months of age. There were no significant differences between the intervention and comparison groups in any of the parent, child, or home visitor demographic characteristics..

Table 1 Participant characteristics: home visitors and parent participants
Full size table

Home-Visit Observation Rating Scales

The seven HOVRS-A+ scales are scored on a 7-point scale; higher scores indicate a higher quality home visit. In the present study, the overall scores for the Home Visitor Practices domain for home visitors in the intervention group ranged from “adequate” (score of 3) to “good” (score of 5) (M = 3.79, SD = 0.97), while the comparison group scored significantly lower (M = 2.77, SD = 1.04), t(39) = 3.04, p = 0.004. The overall scores for the Family Engagement domain were “good” for both the intervention (M = 5.02, SD = 1.39) and comparison (M = 4.48, SD = 1.38) groups. These overall scores are like those in previous studies that used the HOVRS-A+. For example, in the Illinois Prevention Initiative Monitoring study, Korfmacher et al. (2012) found that the Home Visitor Practices were within the adequate-to-good quality range (M = 3.71, SD = 1.03), and Family Engagement was “good” (M = 4.59, SD = 1.11). Table 2 displays the descriptive statistics for the scales at the four data collection times.

Table 2 Mean home visit rating scale (HOVRS-A+) scores by group (N = 42)
Full size table

Intervention Effects

The intervention had a significant positive effect on one of the seven HOVRS-A+ scales, the home visitor responsiveness to family scale. Home visitors in the intervention group scored significantly higher on the home visitor responsiveness to family scale than those in the comparison group. This indicates that the home visitors who received the intervention more frequently engaged in responsive behaviors during the visit—such as asking open-ended questions and commenting on parent or child strengths—and more frequently elicited input on the content and activities of the home visit from the parent (Roggman et al., 2010). In addition, the estimated fixed effect of time on the overall score of responsiveness to family scale was positive, indicating that home visitors’ scores on this scale tended to increase over time.

The IECMHC scale developed for this study was analyzed to test for group differences in scores over time. The interaction term in the two-way ANOVA model was significant, F(24, 40) = 2.31, p = 0.044, ηp2 (partial eta squared) = 0.78. The home visitors in the intervention group had scores on the IECMHC scale items in the HOVRS-A+ that increased at a greater rate, on average, than the scores of the home visitors in the comparison group. Thus, home visitors who received the Illinois model of IECMHC engaged in more behaviors that promoted parenting self-efficacy, encouraged positive parenting behavior, and facilitated responsive parent–child interactions, and the parents in these visits engaged in more positive interactions with their children.

Home visitors in the intervention group scored higher, on average, than those in the comparison group on the Home Visitor Facilitation of Parent–Child Interaction scale, although this effect did not reach statistical significance (p = 0.08). This trend suggests that home visitors in the intervention group facilitated and promoted positive parent–child interactions and encouraged the parent’s leadership in interactions during the visit, but future research with larger samples are needed to detect effects.

No intervention effects were found for the other five HOVRS-A+ scales. See Table 3 for results of the LMM analyses for the final model for each scale.

Table 3 Home visit observations: LMM analysis results (N = 7 home visitors, 42 families)
Full size table

Discussion

This study examined the outcomes of the Illinois Model of IECMHC in home visiting programs. In particular, the study sought to understand the effects of the model on home visitor practice, relationships with families, and family engagement in home visits. This study was the first IECMHC evaluation to use a matched-comparison group design to examine home visiting outcomes. We found that home visitors who received IECMHC were more responsive to families and promoted more parenting self-efficacy, positive parenting behavior, and responsive parent–child interactions, compared to those in the comparison group.

Responsiveness, as measured by the HOVRS-A+ scale, reflects how a home visitor plans and executes the visit with parent input, adapts activities as needed, and identifies family strengths (Roggman et al., 2010). Home visitor responsiveness is an indicator of quality because it helps promote parenting behaviors that support child development, one of the primary goals of home visiting (Roggman et al., 2008). When home visitors build on the family’s strengths, it increases parent confidence and parenting self-efficacy, which is associated with positive parenting behaviors, such as sensitivity and responsiveness (Coleman & Karraker, 2003; Leerkes & Crockenberg, 2002; Papoušek et al., 1998; Teti & Gelfand, 1991). Additionally, home visitor responsiveness is associated with providing an individualized approach to families (Korfmacher et al., 2019), which allows for flexibility in working with families who are hard to reach and engage (Stargel et al., 2018), such as homeless families (Bassuk et al., 2010) and depressed mothers (Tandon et al., 2020). The flexibility of the Illinois Model of IECMHC allows consultants to use an individualized approach in the way they support staff, modeling for staff how they can do the same with families to provide support based on the families’ needs.

Home visitors’ scores also increased over time more than the comparison groups’ scores on the IECMHC scale, developed for this study. The 13 items on this scale are theoretically linked with the expected outcomes of IECMHC, which include home visitors being more responsive to families, having a strong relationship with families, facilitating more developmentally appropriate parent–child interactions, following parent and child cues more, and improved parent–child interaction during visits. Each of those domains is linked with positive child and family outcomes. For example, positive home visitor–family relationships are associated with sustained parent enrollment in home visiting programs and increased parent participation during home visits (Korfmacher et al., 2007; Roggman et al., 2001; Roggman et al., 2008; Woolfolk & Unger, 2009). Research also suggests that home visitors who can build positive, trusting relationships with families more effectively strengthen parenting skills and home visiting quality (Gomby et al., 1993; Heaman et al., 2007; Korfmacher et al., 2007; Riley et al., 2008; Saïas et al., 2016).

Finally, the study also suggests that the HOVRS-A+ is a potentially viable tool for measuring the effects of IECMHC in home visiting, although additional research is needed. The HOVRS-A+ has been widely used to measure home visit quality (Korfmacher et al., 2019; Roggman et al., 2016; Vogel et al., 2015) but has not been used to measure the effects of mental health consultation. The domains measured by the HOVRS-A+ are important indicators of home visit quality, as well as outcomes that IECMHC aims to strengthen. Future research is critical to determine whether the IECMHC scale is appropriate when evaluating IECMHC initiatives, including an analysis of its psychometric properties and replicating findings using additional home-visit observations.

Limitations

One significant limitation is that our sample size was insufficient to isolate home visitor survey data from other staff data in the study, leaving us unable to confidently determine the mechanism of change. Because of the limited number of eligible home visiting programs for this pilot study, our sample was small. Findings imply that the improvement in home visitor quality was due to an improvement in reflective capacity (Spielberger et al., 2022), but because of the small home visitor sample, we were unable to confirm this. Other interventions may offer insight into the mechanism through which IECMHC may improve home visitor practice, such as the Facilitating Attuned INteractions (FAN; Gilkerson et al., 2012). The FAN is a model based in an infant mental health approach with the goal of promoting attunement in relationships between home visitors and parents and fostering staff reflective capacity. In the Fussy Baby Network cross-model FAN evaluation, home visitors improved in reading parents’ cues for engagement, matching interactions based on these cues, maintaining a focus on parenting, exploring parents’ concerns together, and encouraging parents to lead visits (Spielberger et al., 2019). Consultants in the study were trained to use the FAN in their work, which might have improved home visitors’ relationships with families and reflective capacity, but additional research is needed.

Another limitation was that the sample size was not sufficient to conduct meaningful subgroup analyses while maintaining confidentiality. Effect sizes for the kinds of outcomes targeted and measured in IECMHC tend to be small, and these data are clustered in levels, requiring large sample sizes to have the power to detect these effects. Although we conducted power analyses to determine the necessary sample sizes for a matched comparison design, we were limited by the lack of comparable studies in the literature that reported findings on staff and family outcomes in IECMHC in home visiting.

Given the lack of eligible programs in each program type, geographic region, and community type, we used a matched-comparison group design in the evaluation to measure change that could be attributed to the intervention, which is widely considered to be a rigorous study design when it is not possible to conduct a randomized control trial (see Hanita et al., 2017). One limitation of this design was the inability to match programs on every relevant staff or program characteristic before implementation began.

Another potential limitation was that the home visiting programs in the comparison group were operating as “business as usual,” and some staff may have had access to mental health consultation during the study. They were not receiving the Illinois Model, but access to other forms of IECMHC may have masked measurable change in the intervention group.

Conclusion and Implications for the Field

IECMHC shows promise for improving home visit quality, yet additional research is needed to understand how it affects quality and child and family outcomes and how to accurately measure its impact. The HOVRS-A+ scale created for this study shows promise and should be tested in future research on IECMHC in home visiting. Although the current study was unable to directly explore the mechanism of change, the broader study findings suggest that improved home visitor reflective capacity may have been a key factor (Spielberger et al., 2022). We also do not know if certain consultant activities or support—for example, opportunities for home visitors to process visits with consultants, consultants modeling practices that home visitors could apply in their work with families, and trainings in mental health topics—impacted certain home visitor practices. Further rigorous study of IECMHC and the key components could help determine how and where to implement IECMHC to support positive changes in children and families. Home visiting programs can improve the health and well-being of children and families if well implemented, yet identifying ways to increase effectiveness is essential. IECMHC should be considered as one possible method of improving the quality of home visiting services.