The 21st century has ushered in an unprecedented proliferation of innovative tools and technologies designed to improve the quality of mental health services. Less than 30 years ago, treatment manuals were regarded as the primary innovation that would revolutionize services (Luborsky & DeRubeis, 1984; Wilson, 1996). Today, innovations abound in the form of mobile mental health applications, telehealth platforms, transdiagnostic therapies, and electronic health records (Barlow et al., 2016; Chandrashekar, 2018; von Esenwein & Druss, 2014; Zhou et al., 2020). Despite their potential to improve service quality, many innovations struggle to cross the “stagnation chasm,” or the period between an innovation’s initial introduction and its widespread use by individuals within a system (Deglmeier & Greco, 2018). Scholars have argued that efforts to encourage the use of innovations are thwarted, in part, by the field’s disproportionate focus on individual and organizational determinants of implementation outcomes (Lyon & Bruns, 2019) and limited understanding of how determinants influence desirable outcomes (Lewis et al., 2018).

Reviews of implementation research have found only a handful of studies that examined determinants of mental health professionals’ intentions to use innovations (i.e., use intentionsFootnote 1; Eccles et al., 2006; Godin et al., 2008; Perkins et al., 2007). In a large-scale review of implementation outcome measures, Lewis and Colleagues (2016) found that only 24 out of 105 measures assessed adoption, of which only a fraction defined adoption as relating to intentions. Such results are surprising considering use intentions have been posited as a primary implementation outcome (Proctor et al., 2011), linking contextual variables with individual behavior (e.g., organizational structure, policies; Williams and Glisson, 2014). Intentions also play a primary role in nearly all theories of behavior change (Fishbein et al., 2001), and experimental studies support the causal pathway between behavior change interventions, intentions, and subsequent behavior (Webb & Sheeran, 2006).

Studies that examine determinants of those intentions typically focus on individual and/or organizational characteristics, whereby the individual is the person providing services and the organization represents the agency overseeing service delivery (Damschroder et al., 2009). This disproportionate attention is reflected in the field’s measurement–a systematic review of more than 420 implementation assessment tools found 98 measures that assessed individual characteristics, 90 measures for organizational characteristics, and only 19 measures for innovation characteristics (Lewis et al., 2016). However, the empirical examination of individual and/or organizational characteristics has not yet yielded substantial leaps in the understanding of use intentions. Individual characteristics such as general attitudes, demographics, and professional background are widely researched, yet their effects on use intentions are relatively small (Chaudoir et al., 2013; Squires et al., 2011). For example, two studies found general attitudes towards evidence-based practice accounted for approximately 2–15% of the variance in therapists’ intentions (Hill et al., 2021; Mah et al., 2020). Another study found substance use workers’ clinical experience and demographic characteristics explained less than 1% of the variance in use intentions (Kelly et al., 2012). Similarly, organizational climate (e.g., workers’ perceptions of organizational mission) and culture (e.g., organizational expectations about work) are also frequently examined in the implementation literature (Allen et al., 2017). However, two studies estimated that climate accounted for roughly 0–15% of the variability in intentions (Simpson et al., 2007; Williams, 2015). Studies of use behavior have found similarly null to moderate effect sizes for individual and organizational characteristics (e.g., theoretical orientation, clinical experience, organizational culture; Beidas et al., 2017; Friedmann et al., 2007; Henderson et al., 2007; Nelson and Steele, 2007) that may even be redundant with each other (Beidas et al., 2015). Although these studies represent a limited sample of effect sizes, they nonetheless illustrate the need to consider the influence of other variables on use intentions.

Although innovation characteristics are widely accepted as determinants of implementation outcomes across multiple implementation theories (e.g., Consolidated Framework for Implementation Research; Damschroder et al., 2009; Nilsen, 2015), these characteristics are not measured as often as are individual and organizational characteristics in research (Lewis et al., 2016). Yet, an accumulating body of research points to innovation characteristics, in the form of individuals’ beliefs about an innovation, as a promising consideration in mental health implementation efforts. For example, Reding and colleagues (2014) found therapists’ beliefs about appeal and limitations varied by the type of evidence-based treatment, however, only beliefs about appeal were positively associated with self-reported use of specific treatments. Another study by Borntrager and colleagues (2009) established how evidence-based practices organized in a modular format were rated by therapists as more favorable compared with a manualized format. Studies explicitly examining use intentions have found that the strength of therapists’ and teachers’ use intentions depended on the type of practice (e.g., exposure, visual schedules; Fishman et al., 2018; Wolk et al., 2019) and that stronger intentions were associated with future use (Fishman et al., 2018, 2020). These studies show that individuals distinguish among the features of discrete mental health technologies and that individual beliefs are related to intentions and behavior. Whereas intuition might lead one to assume that most mental health technologies are relatively similar, these studies suggest that design matters. Such results may not be surprising since some implementation theories posit a direct relationship between the perceived attributes of an innovation and intentions to use the innovation (e.g., Rogers’ Diffusion of Innovations; Rogers, 2003). Yet, there is still more to discover, including which beliefs matter or how beliefs about an innovation fit into a broader implementation framework alongside individual and organizational characteristicsFootnote 2.

Since the 1940s, psychological research has used rigorous scientific methods to examine how humans physically interact with technology as well as their interpretations of their experiences; which design features represent a mismatch to the typical user or designated task; and how to build solutions that are workable, efficient, and that support an array of user capabilities (Chapanis et al., 1949; Proctor et al., 2021). The field of human factors engineering focuses on optimizing the design of objects for human use, with the central premise that human performance is inextricably linked to the design of the proximal technology or environment (Chapanis et al., 1949). Other fields have also amassed significant scientific evidence about the role of design on human use and performance. Research from the field of information technologies provides ideas about which specific user beliefs about innovations may be meaningful in mental and behavioral health. According to the Unified Theory of Acceptance and Use of Technology (UTAUT), an individual’s use intentions and behavior are determined by beliefs that the innovation is (1) likely to produce desirable outcomes, (2) easy to use, (3) valuable to important others, (4) feasible given the resources at their disposal, (5) enjoyable to use, (6) offers value above and beyond its price, and (7) routine to use (Venkatesh et al., 2003, 2012). The UTAUT explained up to 74% of the variability in use intentions and 52% of the variability in use behavior in the original studies (Venkatesh et al., 2003, 2012). In comparison, constructs outlined by the Theory of Planned Behavior and Diffusion of Innovations Theory (e.g., self-efficacy, Ajzen, 1991; relative advantage, Rogers, 2003) accounted for less than 40% of the variability in intentions and behavior within the same study (Venkatesh et al., 2003). Similar findings have been found across different innovations, individuals, and organizational settings (e.g., Brown et al., 2010; Shibl et al., 2013; see Venkatesh et al., 2016 for review). It is important to note the UTAUT includes constructs from well-established behavior change theories (e.g., intentions, subjective norms; Ajzen, 1991) with several important theoretical distinctions (e.g., complexity vs. ease of use; Venkatesh, 2003). Thus, the theory offers additional beliefs about an innovation that can be easily incorporated with models examined in implementation research.

The UTAUT has two important advantages for implementation research–namely, specificity of measurement and the beliefs (versus attitudes) construct. Implementation scholars are increasingly advocating for measures to reference a specific innovation of interest rather than a general group of innovations (e.g., exposure vs. evidence-based practices; Fishman et al., 2021), typically in the context of general attitudes towards evidence-based treatments or practices (e.g., Aarons et al., 2004). A similar emphasis on measurement specificity is embedded in the UTAUT and extended to the beliefs construct. The UTAUT draws on Davis’s (1993) Technology Acceptance Model (TAM) and the TAM’s own influences from Fishbein & Ajzen’s (1975) Theory of Reasoned Action (Chuttur, 2009). According to the TAM and TRA, attitudes are defined as the positive or negative feelings towards a behavior (in this case, using an innovation), which are distinct from the behavioral or normative beliefs about the consequences of performing a behavior. Whereas attitudes are a general predisposition towards an innovation, beliefs are the underlying cognitions that influence attitudes (Fishbein & Ajzen, 1975; Taylor & Todd, 1995). Distinguishing between attitudes and beliefs is critical given the wide variability in the conceptualization and measurement of “attitudes” in implementation research (Fishman et al., 2021). Consequently, the UTAUT has potential both as a specific measurement approach and a means to provide greater conceptual clarity.

To our knowledge, no published studies have explored the UTAUT’s potential contributions to the mental health implementation literature. Several studies of mobile mental health applications have applied the UTAUT (Damerau et al., 2021; Hennemann et al., 2018; Mitchell et al., 2021). However, it is unclear whether different results would be found with innovations designed for mental health professionals and/or the complex systems in which these innovations are implemented. Additionally, to our knowledge, no studies have compared the predictive utility of the UTAUT (or any other well-defined construct reflecting innovation beliefs) with other individual and organizational characteristics. Comparison studies have the potential to refine the field’s conceptualizations about whether individual, organizational, and innovation characteristics are more proximal or distal determinants of individual intentions (e.g., Rogers, 2003; Williams and Glisson, 2014). Advancing the field’s ability to improve implementation outcomes will require differentiating proximal variables that exert direct, immediate impacts on intentions versus distal variables comprised of a chain of indirect influences. Such work necessarily entails identifying variables with strong statistical relationships with implementation outcomes and testing theoretically-informed models of causal processes (e.g., mediation, directed acyclic graphs).

The Present Study and Aims

The purpose of this study was to examine multiple determinants of mental health therapists’ intentions to use innovations within a single study. Intentions were chosen as the primary outcome given their central role in linking behavior change interventions with individual behavior change (Webb & Sheeran, 2006). We evaluated the role of individual, organizational, and innovation characteristics as more proximal versus distal determinants of intentions with a principal focus on innovation characteristics. Our three aims were to (1) examine to what extent innovation characteristics were more proximal versus distal predictors of intentions relative to individual and organizational characteristics; (2) evaluate the magnitude of the relationship between innovation characteristics and intentions when controlling for individual and organizational characteristics; and (3) test theoretically-informed mediation models from the information technologies literature regarding whether therapists’ beliefs about an innovation explained differences in intention between two innovation types–the Reaching Families Engagement System and Practice Guidelines. Aims 1 and 2 provided evidence for the proximal role of innovation characteristics and also established several preconditions for the subsequent mediation analyses in Aim 3.

Method

Data were collected during a multi-site cluster randomized trial that tested the effectiveness of two types of clinical decision-making innovations to promote youth and family treatment engagement in mental health services (Chorpita & Becker 2017-2022). Supervisors and therapists were recruited from two mental health organizations: the Los Angeles Unified School District (LAUSD) or the South Carolina Department of Mental Health (SCDMH). Supervisory groups consisting of a supervisor and multiple therapists were randomly assigned to one of two experimental conditions: (1) Reaching Families Engagement System (RFES) which included a set of knowledge resources that guided the assessment of engagement as well as the selection and delivery of engagement practices or (2) Practice Guidelines (PG) which included a list of practices and their descriptions that represented a traditional resource for guiding the selection and delivery of engagement practices (see Becker et al., 2019 for a detailed description of these conditions). The RFES and PG constitute the two innovations examined in this study. Supervisory dyads utilized the RFES or PG during supervision and completed measures about their respective individual, organizational, and innovation characteristics. All study procedures were approved by the institutional review boards of the University of California, Los Angeles, the University of South Carolina, and the participating school mental health agencies. Written informed consent was obtained and study procedures were executed in accordance with local IRB requirements. The present study was not preregistered.

Participants

Participants consisted of 95 therapists and 28 supervisors across Los Angeles and South Carolina. The number of therapists working under each supervisor ranged from 1 to 6, with an average of 3.39 (SD = 1.42) therapists per supervisor. Approximately 93% (n = 26) of supervisory groups consisted of two or more therapists. On average, participants were 42 years old (SD = 10.8) at the time of data collection, of which 92% were female. Therapists identified as Latinx (41%), Black (39%), White (16%), Asian (4.4%), and Middle Eastern (0.1%). Therapists and supervisors from both organizations (i.e., LAUSD, SCDMH) provide mental health services on public school campuses as well as in local mental health clinics.

Measures

Individual Characteristics

Professional Background. Therapists and supervisors completed a demographics and training background questionnaire. Therapists reported on years of clinical experience, number of active client cases, and how often they experience feelings of professional burnout on a 5-point scale from 0 never to 5 all of the time. Supervisors reported their years of clinical experience, years of supervision experience, and number of active supervision cases.

Attitudes Towards Evidence-Based Practices. Therapists’ general attitudes towards evidence-based practices were assessed using four subscales of the Evidence-based Practice Attitude Scale 50-item (EBPAS-50; Aarons et al., 2012): (1) appeal (i.e., likelihood of adopting an evidence-based practice if it were intuitively appealing), (2) requirements (i.e., likelihood of adopting an evidence-based practice if it were required), (3) openness (i.e., therapist openness to trying new interventions), and (4) divergence (i.e., therapist beliefs that evidence-based practices were less useful than clinical experience). The Appeal, Requirements, Openness, and Divergence subscales correspond to the four original scales on the EBPAS-15 (Aarons et al., 2010). Since this study sought to contrast general attitude measurement with innovation-specific beliefs rather than provide an exhaustive examination of all general attitudes towards evidence-based practice, the remaining eight subscales (e.g., Limitations, Fit) from the extended EBPAS-50 were omitted from analyses to reduce model complexity. Subscale scores were calculated by averaging the four items within each scale, which were rated on a 5-point Likert scale from 0 not at all to 4 to a very great extent. Cronbach’s alphas for the four EBPAS subscales fell in the acceptable range from 0.64 to 0.88, with the lowest reliability found for the Divergence subscale. Reliability estimates were consistent with those found in a large-scale psychometric study by Aarons and colleagues (2010) who found alpha reliabilities that ranged from 0.66 to 0.91. Evidence for the construct and convergent validity of the EBPAS was also supported in the original scale development studies (Aarons, 2004; Aarons et al., 2007).

Organizational CharacteristicsFootnote 3

To assess perceptions of organizational climate, therapists completed 30 items across the six organizational climate subscales of the Texas Christian University Organizational Readiness for Change: Treatment Staff Version (TCU ORC-S; Institute of Behavioral Research, 2003): Mission, Cohesion, Autonomy, Communication, Stress, and Change. These six subscales assessed therapist perceptions of the degree to which they were aware of the organization’s mission or goals, they cooperated and trusted each other, they were granted decision-making authority, their suggestions were heard and valued by management, they experienced excessive work strain or overload, or they believed management was interested in and adaptable to novel changes. All items were scored on a 5-point Likert scale ranging from 1 strongly disagree to 5 strongly agree and subscale scores were calculated by averaging items within each scale. Cronbach’s alphas were 0.80 or greater for the Cohesion, Communication, and Stress subscales. The Mission, Autonomy, and Change subscales demonstrated reliabilities ranging from 0.52 to 0.66. In a psychometric evaluation of the TCU-ORC-S scales, Lehman and colleagues (2002) found scale reliabilities for program staff of 0.57 to 0.84. Support for the dimensionality of the six subscales was found in the same study.

Innovation Characteristics and Use Intentions

Therapists completed an adapted version of the UTAUT-2 10-item measure based on the extended UTAUT model (UTAUT-2; Venkatesh et al., 2012). We created this 10-item version to represent six of the eight constructs from the extended UTAUT. Item selection was informed by face validity and concern about participant burden. Constructs measuring Price Value and Facilitating Conditions were omitted because therapists did not need personal finances or supplementary resources to utilize the innovations. The adapted measure assessed therapists’ beliefs about the RFES or PG along five dimensions: (1) performance expectancy (i.e., beliefs that the innovation would produce desirable outcomes), (2) effort expectancy (i.e., beliefs that the innovation was easy to use), (3) social influence (i.e., beliefs that the innovation was valuable to important others), (4) hedonic motivation (i.e., beliefs that the innovation was enjoyable to use), and (5) habit (i.e., beliefs that using the innovation had become routine). The measure also assessed use intentions (i.e., the degree to which therapists planned to use the innovation in the future), which served as the primary dependent variable for this study.

The administration of the UTAUT occurred via an online survey platform that allowed us to tailor the survey presentation according to each study condition (i.e., RFES, PG). Wording from the original items was retained whenever possible, with the exception of the term “mobile internet” that was the target technology in the UTAUT-2 (Venkatesh et al., 2012). To improve comprehension and orient therapists to respond about their beliefs about their study condition-specific innovation (either RFES or PG), the survey showed a picture of the respondent’s specific innovation ahead of administering items. Then, instead of including the term “mobile internet,” each item presented a familiar term that had been used to communicate about each innovation throughout the study. These procedures were consistent with previous adaptations of the UTAUT measure (Knudsen et al., 2021). A full list of the original UTAUT-2 items is documented in Venkatesh and colleagues (2012).

All items were rated on a 7-point Likert scale from 1 strongly disagree to 7 strongly agree. Performance and effort expectancies were assessed by three items each and scale scores were calculated by averaging items within the scale. The remaining constructs were measured with one item and utilized the raw item scores. Alpha reliabilities for the Performance and Effort Expectancy scales were 0.94 and 0.93, respectively. Reliability and validity of the UTAUT is documented extensively in the original scale development studies (Venkatesh et al., 2003, 2012). In addition to finding internal consistency reliabilities and factor loadings that exceed 0.80 for most subscales, these studies found strong evidence for the convergent and discriminant validity of the UTAUT. One study evaluated reliability of the UTAUT in a mental health context and estimated Cronbach’s alpha coefficients that ranged from 0.81 to 0.96 (Damerau et al., 2021).

Procedure

Therapists and supervisors completed the professional background questionnaire prior to receiving training with the RFES or PG. The TCU ORC-S was completed mid-study and the EBPAS-50 and UTAUT were completed by therapists at the end of the study. This study design allowed ample opportunity for therapists to use their respective innovation.

Data Analytic Plan

All analyses were performed using R Statistical Software (v.4.2.1; R Core Team, 2022). To examine to what extent innovation characteristics were more proximal versus distal predictors of intentions relative to individual and organizational characteristics (Aim 1), a series of multilevel models were constructed to estimate the effect size and statistical significance of each individual, organizational, and innovation characteristic. Further multilevel models were constructed to evaluate the magnitude of the relationship between innovation characteristics and use intentions when controlling for individual and organizational characteristics (Aim 2). Analyses for Aim 1 and Aim 2 were designed to examine whether effect sizes for innovation characteristics were larger in magnitude and statistically significant compared with individual and organizational characteristics, which would provide empirical evidence for the proximal and potential mediating role of innovation characteristics (Aim 3).

Prior to these analyses, an unconditional model with random intercepts for supervisors and service sites was estimated to determine proportion of variance explained at each level. Intra-class correlation coefficients indicated that 28% of the variance in use intentions was explained by supervisors and 6% of the variance by service sites. The random intercept for service sites was omitted from all subsequent models since it explained a relatively small proportion of variance in the outcome variable. All models were estimated using the lme4 and lmerTest R packages (Bates et al., 2014; Kuznetsova et al., 2017). All models utilized restricted maximum likelihood estimation to adjust for small sample bias (McNeish, 2017). Data from the UTAUT scales were complete. Missing data ranged between 0 and 11% on items from the professional background questionnaire. EBPAS and TCU ORC-S subscale scores were missing for 0–2% of participants. Missing data were imputed using the mice R package with n = 10 datasets and parameter estimates pooled according to Rubin’s rules (van Buuren & Groothuis-Oudshoorn, 2011).

Aim 1: Proximal Versus Distal Role of Individual, Organizational, and Innovation Characteristics

Separate multilevel models were constructed with each measure of individual (i.e., therapist background, evidence-based practice attitudes, supervisor background), organizational (i.e., organizational climate), and innovation characteristics (i.e., UTAUT) predicting use intentions. Models were also estimated for innovation type (RFES or PG) and site (Los Angeles or South Carolina). All predictors were estimated as fixed effects. Level-1 predictors were group-mean centered, level-2 predictors grand-mean centered, and the group means of level-1 variables were entered at level-2. Level-1 predictors included the UTAUT, therapist background, evidence-based practice attitudes, and organizational climate. Level-2 predictors included supervisor background, innovation type, and study site. Using this approach, estimates of within- and between-group regression coefficients and R2 effect sizes could be obtained for all level-1 predictors (Enders & Tofighi, 2007). Within-group effects represented the effect of a variable on therapists’ intentions when therapists had higher or lower values relative to other therapists in their supervisory group. Between-group effects are interpreted as the effect of a variable on therapists’ intentions when their supervisory group had higher or lower values relative to other supervisory groups. Thus, within- and between-group regression coefficients and effect sizes estimate the independent influence of variables at the individual- and group-level, respectivelyFootnote 4. Three R2 measures were calculated with the r2mlm R package (Shaw et al., 2023) to capture total variance in use intentions explained by level-1 fixed effects (R2f1), total variance explained by level-2 fixed effects (R2f2), and total variance explained by both level-1 and level-2 fixed effects (R2f; Rights and Sterba, 2019).

Aim 2: Magnitude of Innovation Characteristics Controlling for Individual and Organizational Characteristics

A subset of variables from the first stage of analyses were identified as meaningful predictors of use intentions. Individual and organizational characteristics with p < .10 were collectively entered into a single model to create a set of covariates that could be compared to innovation characteristics. Innovation characteristics with p < .10 were also estimated in a single model. Finally, a combined model was estimated with the subset of individual, organizational, and innovation characteristics. R2 measures were calculated for each model along with change in R2 between models to ascertain the incremental predictive value of each. Parameter estimates and patterns of statistical significance between models were also examined to assess the unique effect of each predictor while controlling for individual, organizational, or innovation characteristics.

Aim 3: Mediating Role of Innovation Characteristics

Results from Aim 2 laid the foundation for mediation analyses to test whether therapists’ beliefs about an innovation explained differences in intention between the RFES and PG innovation types using two theoretically-informed models from the information technologies literature. In Aim 2 analyses, innovation type evidenced a statistically significant relationship with use intentions that was non-significant after controlling for therapists’ reports of innovation characteristics on the UTAUT, and the coefficient for innovation type was reduced by a factor of five. These results were empirically consistent with a mediational process (Baron & Kenny, 1986). Given that the design features of the innovations differed between the two innovations, results were also theoretically consistent with previous research that performance and effort expectancies mediate the effect of technology characteristics on users’ intentions (Brown et al., 2010; Rahi et al., 2019; Venkatesh & Bala, 2008).

All analyses were performed with the lavaan R package (Rosseel, 2012). Prior to the mediational analyses, discriminant validity between the performance and effort expectancy items was assessed via confirmatory factor analyses. A conventional factorial validity approach was utilized (Brown, 2015), which entailed testing a hypothesized two-factor model against an alternative model representing both subscales as a single dimension in a one-factor model. Model fit was determined with the model χ2, differential χ2 test, the comparative fit index (CFI; Bentler, 1990), root mean square error of approximation (RMSEA; Steiger, 1990), and standardized root mean square residual (SRMR; Jörsekog, 1993). Discriminant validity of the single use intentions item was assessed by estimating a confirmatory factor model with use intentions cross loaded on the performance and effort expectancy subscales. Akaike’s information criteria (AIC; Akaike, 1973) was included as an additional fit index to assess the relative degree of misfit introduced by allowing the use intentions item to load on the performance and effort subscales (Iacobucci, 2010). All factor models utilized maximum likelihood estimation.

Aim 3 utilized a model comparison approach to test two mediational pathways hypothesized in the information technologies literature (Venkatesh et al., 2003; Venkatesh & Bala, 2008). Figure 1 depicts a sequential mediational process theorized by the Technology Acceptance Model 3 (TAM3) where the effect of innovation type on use intentions is mediated by effort first, and then performance expectancies (Venkatesh & Bala, 2008). Figure 2 depicts a parallel mediational process outlined by the UTAUT in which the effect of innovation type is mediated by performance and effort expectancies simultaneously (Venkatesh et al., 2003). Both mediation models were fit using maximum likelihood estimation with robust test statistics and bootstrapped standard errors. Innovation type was dummy coded such that therapists who used the RFES were assigned a value of 1. Average scale scores were utilized for performance and effort expectancies to minimize the complexity of adding latent variables to the model. Model fit was determined with the model χ2, CFI, SRMR, and RMSEA. Parameter estimates were interpreted only for the model that demonstrated superior fit, both overall and relative to the other model. Mediated indirect effects were estimated using the product of coefficients approach. Estimates of indirect effects were tested by constructing 95% confidence intervals using the adjusted bootstrap percentile method.

Fig. 1
figure 1

Mediation Model 1

Full size image
Fig. 2
figure 2

Mediation Model 2

Full size image

Results

Aim 1: Proximal Versus Distal Role of Individual, Organizational, and Innovation Characteristics

Table 1 contains effect size and parameter estimates for each measure of individual, organizational, and innovation characteristics. Total variability (R2f) in use intentions explained by measures of individual characteristics ranged from 9 to 11%. Organizational characteristics, as measured by the organizational climate scales, accounted for 22% of the variability in intentions. Innovation type and site accounted for 9% and 10% of the variability in intentions, respectively.

Table 1 Effect Sizes and Parameter Estimates for Each Measure of Individual, Organizational, and Innovation Characteristics
Full size table

Innovation characteristics explained 76% of the variance in intentions–more than three times that of any other individual or organizational measure.

Between- and within-group effects differed for several measures. Nearly all the variance explained by organizational climate was attributable to the between-group effect, R2f2 = 0.19, and similarly for attitudes towards evidence-based practice, R2f2 = 0.10. Only the between-group effect of the organizational climate autonomy subscale (B = 1.37, p = .013) and EBPAS openness subscale (B = 1.10, p = .013) were significantly associated with intentions. Specifically, supervisory groups who collectively felt their organization granted them decision-making authority and were open to using new types of interventions had therapists with stronger intentions to use the innovation. However, therapists’ individual feelings of autonomy or openness were not statistically associated with stronger intentions. Notably, the UTAUT subscales measuring hedonic motivation (B = 0.35, p < .01) and habit (B = 0.38, p < .001) were only significant at the within-group level. Therapists’ beliefs that the innovation was enjoyable and routine to use were associated with stronger intentions but supervisory groups’ collective beliefs were not. Performance expectancies were the only UTAUT subscale statistically significant at both the therapist and supervisory group level.

Aim 2: Magnitude of Innovation Characteristics Controlling for Individual and Organizational Characteristics

Table 2 contains effect sizes and parameter estimates for the subset of individual, organizational, and innovation characteristics utilized in model comparisons. The covariates model with the individual and organizational variables, plus innovation type and site, explained 29% of the total variability in therapists’ use intentions, R2f = 0.29. Innovation characteristics explained 75% of the variance in use intentions, R2f = 0.75, which was a 46% increase compared to the model with only covariates, ΔR2f = 0.46. Combining all variables from the covariates and innovation characteristics models into a single model resulted in no increase on any of the R2 measures.

Table 2 Effect Sizes and Parameter Estimates for Covariates, UTAUT, and Combined Model
Full size table

Two individual (i.e., openness and supervisor experience) and one organizational (i.e., autonomy) variable previously associated with intentions were no longer significant in the covariates model. Only therapists’ client caseload (B = 0.02, p = .020) and innovation type (B = 0.73, p = .023) remained statistically significant in the covariates model. Neither was significant in the combined model, and the magnitude of regression coefficients decreased by as much as a factor of five compared to the covariates model (e.g., B = 0.73 vs. B = 0.13 for innovation type). In contrast, all innovation characteristics remained significant in the combined model and regression coefficients decreased by a factor of 1.15 at most. Variance inflation factors were less than 3 across all models and fell below suggested thresholds that indicate problematic multicollinearity among predictor variables (O’Brien, 2007).

Aim 3: Mediating Role of Innovation Characteristics

Table 3 contains results from the confirmatory factor models to establish discriminant validity of the mediator and criterion variables. The two-factor model for performance and effort expectancy items fit uniformly better than a one-factor model on all fit indices, as indicated by a χ2 difference of 164.09, with 1 degree of freedom, p < .001. A two-factor model with the single use intentions item cross loaded on both factors also demonstrated worse fit on all indices, and factor loadings for the intentions item ranged from 0.31 to 0.55. The two-factor model met suggested thresholds for good fit on all indices (i.e., CFI < 0.95, SRMR < 0.06, RMSEA < 0.08; Hu and Bentler, 1999) and the χ2 test of overall model fit was non-significant, which further supported these scales appeared to have measured separate constructs. Standardized factor loadings ranged from 0.85 to 0.95 for the performance factor and 0.83 to 0.98 for the effort factor.

Table 3 Standardized Factor Loadings and Fit Statistics for Hypothesized Two-Factor Factor Model and Alternative Factor Models
Full size table

Results from the mediational path analyses can be seen in Table 4. Model 1, in which the effect of the RFES innovation on intentions was mediated through effort and then performance expectancies, demonstrated better fit on all indices than Model 2 where the effect of the RFES was mediated by performance and effort expectancies simultaneously. Model 1 had a non-significant model χ2 test and met suggested thresholds (i.e., CFI < 0.95, SRMR < 0.06) for good fit on all indices except the RMSEA. However, RMSEA may over-reject correctly specified models when sample size and model degrees of freedom are small (Kenny et al., 2015; Taasoobshirazi & Wang, 2016). Thus, the overall fit of Model 1 provided support for the mediation pathway theorized by the TAM3 (Venkatesh & Bala, 2008). Model 2 did not meet suggested thresholds for good fit on any indices and no support was found for the mediation pathway outlined by the UTAUT.

Table 4 Parameter Estimates for Final Mediation Model 1 and Fit Statistics for Hypothesized Mediation Models 1 and 2
Full size table

Parameter estimates for Model 1 are shown in Fig. 3. Compared with those who utilized the PG innovation, therapists assigned to use the RFES innovation evidenced significantly stronger performance and effort expectancies that were in turn associated with a statistically significant increase in use intentions. The indirect effects of the RFES innovation were positive and statistically significant (i.e., none of the 95% bootstrapped confidence intervals contained zero). Unstandardized path coefficient estimates for indirect effects ranged from 0.199 to 0.410. The direct effect of the RFES innovation was not statistically significant, although the estimated path was greater than zero (B = 0.174, p = .365). All other direct effects were significant at an alpha level of 0.05.

Fig. 3
figure 3

Path Diagram of Results from Mediation Model 1

Full size image

Discussion

Consistent with implementation and behavior change theories, we compared multiple predictors of therapists’ intentions to use innovations in a single study. Although individual and organizational characteristics are well-studied in implementation outcome research, this study built upon decades of research from human factors psychology and related fields to examine how user beliefs about design influence use intentions. Notably, relative to individual and organizational characteristics, innovation characteristics demonstrated the greatest predictive utility for therapists’ use intentions. Aim 1 examined each measure of individual, organizational, and innovation characteristics separately. Therapists’ beliefs about innovation characteristics accounted for more than three times the variability in use intentions than any other measure of individual or organizational characteristics (e.g., attitudes towards evidence-based practice, organizational climate). Aim 2 examined individual, organizational, and innovation characteristics collectively. Individual and organizational characteristics did not explain unique variance in use intentions above and beyond innovation characteristics, evidenced by no increase in R2 when these variables were added to a model with only innovation characteristics. Interestingly, individual and organizational characteristics that were statistically significant in the Aim 1 analyses (e.g., openness towards evidence-based practices, organizational autonomy) were not significant after controlling for innovation characteristics, but innovation characteristics remained statistically significant. These results support the role of innovation characteristics as proximal determinants of intentions. Aim 3 explored whether therapists’ beliefs about an innovation mediated the relationship between innovation type and use intentions. Relative to those therapists who used the PG, those who utilized the RFES had stronger intentions to continue using their innovation, and this study found support for therapists’ performance and effort expectancies as mediators of this relationship. That is, therapists may have had stronger intentions to use the RFES after the study because this innovation was perceived as more effective and easier to use than the PG.

Although this study found similarly null to moderate effect sizes for individual and organizational characteristics (Hill et al., 2021; Kelly et al., 2012; Mah et al., 2020; Simpson et al., 2007; Williams, 2015) that suggest these variables may be more distal determinants of intentions, findings related to specific characteristics were nonetheless noteworthy. In Aim 1 analyses, for example, therapists who reported a greater openness to using new treatments on the EBPAS (Aarons et al., 2010) also reported stronger intentions to continue using their innovation. Because the openness subscale did not reference the RFES or PG, it is possible that a generalized openness towards evidence-based practices is a useful predictor of intentions to use any specific innovation when the innovation is treatment- or evidence-oriented. However, indices of other general attitudes towards evidence-based practice may not be strong predictors, because they may conflate the “evidence-based” attribute of an innovation with its structure or interface (e.g., Borntrager et al., 2009). Therapists who worked with more experienced supervisors also had stronger use intentions. This finding fit with expectations, given that both innovations were designed to support collaborative decision-making between therapists and supervisors during supervision. Other organizational members’ (e.g., supervisors) competencies may have consequences for an individual’s intentions to use an innovation when the innovation requires participation from multiple people. For this study, conceptualizing supervision experience as an “individual” characteristic may be an oversimplification because therapists and supervisors worked as dyads, and therapist perceptions about the quality of their working relationship with their supervisor may be affected by the supervisor’s experience and training (Boyd et al., 2021). Additionally, the fact that therapists who reported their organization granted them greater decision-making autonomy on the organizational climate scales (Institute of Behavioral Research, 2003) reported stronger use intentions was particularly interesting since the innovations in this study were designed for clinical decision-making. The autonomy subscale may have inadvertently measured a facet of implementation climate regarding the extent to which therapists perceived the specific innovation was supported by their organization (Klein & Sorra, 1996). Such an interpretation would be consistent with arguments that implementation climate has greater predictive validity because it emphasizes organizational perceptions that are relevant to implementing a specific innovation and are psychologically proximal to the outcome of interest (Schneider, 1975; Weiner et al., 2011).

In addition to expanding which beliefs about an innovation are related to use intentions, findings related to specific innovation characteristics illustrate the utility of decomposing attitudes into specific beliefs. Taylor and Todd (1995) proposed that determinants of intentions under the theory of planned behavior (i.e., attitudes, perceived behavioral control, subjective norms) were influenced by more specific belief structures relevant to the technology. The relationship between beliefs and attitudes is also congruent with Fishbein and Ajzen’s (1975) Theory of Reasoned Action. Indeed, our study found beliefs that an innovation (1) would produce desirable outcomes, (2) was easy to use, (3) was enjoyable to use, and (4) felt routine to use were all significantly associated with therapists’ intentions and explained more than 75% of the variability in intentions. These findings support previous research that specific beliefs about an innovation are more predictive of intentions (Taylor & Todd, 1995; Davis, 1989), and may account for why the UTAUT outperformed the Theory of Planned Behavior in earlier research (Venkatesh, 2003). Measuring specific beliefs in implementation studies can allow researchers to understand why individuals form more or less favorable attitudes towards innovations and is consistent with decades of human factors research focused on understanding human interaction with the features of a proximal technology or environment (Proctor et al., 2021). Moreover, beliefs provide conceptual links to information technologies studies that examine how external variables (e.g., objective design features of an innovation, training) influence beliefs about the innovation, then subsequently intentions and behavior (e.g., Davis, 1993; Venkatesh and Bala, 2008). Specific belief structures are also more actionable targets for implementation initiatives than an individuals’ affective predisposition towards using an innovation (i.e., attitudes). Indeed, results from Aim 3 suggested performance and effort expectancies may explain why therapists who utilized the RFES had stronger intentions to use their innovation. This finding aligns with a qualitative study conducted among the same sample of participants, which found certain design features of the RFES were perceived as more useful and easier to use than others (Chu et al., 2022). Interestingly, Aim 1 analyses did not find a statistical association between therapists’ beliefs that valued others thought they should use the innovation (i.e., social influence) and use intentions. One explanation is that therapists were not mandated by their organization to continue using the innovations after the study, and thus, there were no social pressures to form greater intentions (Venkatesh & Davis, 2000). Alternatively, it is possible that peers’ opinions about the innovations had an early effect on intentions that diminished as therapists formed their own concrete evaluations of the innovations through their own use.

Finally, it is useful to consider individual, organizational, and innovation characteristics from an ecological perspective. Individual and organizational characteristics did not predict use intentions after controlling for therapists’ beliefs about the innovations. Beliefs may be the most proximal determinants of intentions through which distal individual and organizational determinants transmit their effect. For example, perceptions of organizational decision-making autonomy may have influenced therapists’ beliefs about how easy or difficult the innovations were to use. Such explanations may facilitate greater understanding of how organizational interventions influence individual behavior (e.g., Williams and Glisson, 2014). It is also important to highlight how many characteristics were statistically significant at the between-group level. For example, therapists had stronger use intentions when they belonged to a supervisory group composed of therapists with higher openness to evidence-based treatments. These results support arguments made by implementation researchers to conduct multilevel analyses that examine how characteristics of individuals coalesce at higher levels to influence implementation outcomes at lower levels (e.g., Williams, 2016). Such analyses are especially relevant because implementation efforts may fail if strategies target seemingly individual-level determinants (e.g., individual attitudes) that actually represent characteristics of the larger social ecology (e.g., supervisory groups, clinics). On the other hand, beliefs that the innovations were enjoyable to use and felt routine to use (i.e., hedonic motivation and habit) were significant at the therapist level, which suggests targeting these areas among individuals can promote behavior change even when the broader social ecology does not share the same belief.

Limitations

This study was based on a limited range of innovations, individuals, and mental health settings. It is unclear whether and to what extent these results would generalize across different innovations that are used by supervisors, administrators, or clients. However, results are consistent with other studies that applied the UTAUT to consumer mobile mental health apps (Damerau et al., 2021; Hennemann et al., 2018), and lend further support to the theory’s generalizability with different innovations and individuals. Additionally, the dependent variable for this study was use intentions and it is unknown to what extent use intentions predicted subsequent use behavior. Follow-up surveys of use behavior were not possible due to constraints related to the COVID-19 pandemic. Nonetheless, experimental studies have found that intentions lie along the causal pathway to behavior and are the strongest influence on behavior (Webb & Sheeran, 2006). Additionally, this study examined a limited set of individual and organizational characteristics. It is likely there are other individual (e.g., general self-efficacy) and organizational (e.g., agency size, financial resources) characteristics that are meaningfully related to intentions even after accounting for beliefs about an innovation. Findings from the mediational path analyses should be interpreted with caution given the mediator and criterion variables were assessed at the same time point. Limitations in the data and study design precluded tests of multilevel mediation, bidirectional effects, feedback effects, or unobserved confounders (Ajzen, 2020; Maxwell & Cole, 2007; Williams, 2016). Despite these limitations, these findings supported previous research on performance and effort expectancies (e.g., Chu et al., 2022; Venkatesh and Bala, 2008) and merit more rigorous exploration. These analyses also demonstrated one approach to refining implementation theory by comparing models that hypothesize different causal relationships among the same variables.

Three limitations imposed by modeling and measurement decisions should be considered further. First, it is worth considering whether the organizational climate scales were indeed a characteristic of therapists’ “organization” given our decision to model the climate scales at the individual- and supervisory group-level. Our measures of organizational climate may be conceptually closer to what Glick (1985) described as psychological and subunit climate. While this modeling decision allowed us to compare organizational climate with individual and innovation characteristics, it did so at the expense of a more nuanced understanding of climate (see Schneider et al., 2013 for a more thorough explanation of climate research methods and conceptualizations). Second, our use of the adapted UTAUT scales to measure innovation characteristics and intentions had some constraints. Some constructs were measured with single items and psychometric properties could not be evaluated for all subscales. However, the UTAUT has well-established psychometric properties (Venkatesh et al., 2003, 2012) and we found strong evidence of internal consistency, discriminant validity, and structural validity for the performance and effort expectancy subscales where it was possible to evaluate reliability and validity. Furthermore, the UTAUT was originally developed using items from well-researched, psychometrically sound measures (e.g., Theory of Planned Behavior, Technology Acceptance Model; Venkatesh, 2003) that increases our confidence in its psychometric properties. Common method variance should also be considered as a contributing factor to this study’s findings since the items for intentions and innovation characteristics were from the same measure. This limitation is tempered by Malhotra and colleagues’ (2006) study who evaluated common method variance among performance and effort expectancies with intentions. Common method variance was found to be minimal and adjusting correlations in previous studies for common method variance did not significantly change the published findings. Third, our model included a measure of attitudes about evidence-based treatments broadly (i.e., EBPAS) and a measure of beliefs specific to the innovations providers had used in the present study (i.e., UTAUT). It is possible that the comparison of a broader attitudes measure with a more specific innovation beliefs measure biased the results in favor of the UTAUT. Although this measurement model was intentional and allowed us to answer questions related to general attitudes versus innovation-specific beliefs, future studies could compare measures of specific beliefs and specific attitudes to determine if this conceptual distinction is important for predicting intentions. Despite these limitations in measuring innovation characteristics, this study yielded important evidence that understanding how individuals interact with mental health technologies might be an important area for continued research.

Implications for Future Research

A number of research directions can build on this study’s findings and limitations. Individual, organizational, and innovation characteristics could be assessed in large-scale, longitudinal studies to ascertain how these constructs mutually influence each other. The proliferation of innovative tools and technologies in mental health services presents many opportunities to conduct longitudinal field studies among multiple types of individuals, organizations, and innovations across time. At least one longitudinal study is already underway (e.g., Becker-Haimes et al., 2021), and individuals’ beliefs about an innovation may provide important links between individual difference domains (e.g., motivation), organizational structure (e.g., agency size), organizational processes (e.g., incentives, training), or other contextual characteristics. These studies could adopt a similar model comparison strategy as Aim 3 to reconcile among theories that hypothesize different relationships among the same variables (e.g., UTAUT, TAM3; Venkatesh, 2003; Venkatesh and Bala, 2008) or attribute causality to different determinants (e.g., innovation characteristics versus organizational climate; Rogers, 2003; Williams and Glisson, 2014). Importantly, this work could build upon our study’s limitations by more thoroughly evaluating the UTAUT’s psychometric properties and addressing common method variance with modern statistical and study design approaches (e.g., marker-variable technique; see Tehseen et al., 2017). Another important extension of this work involves examining which objective features of an innovation influence which beliefs. A preponderance of research in human factors, cognitive science, and user-centered design has studied design features and processes that increase the utility and usability of technologies (Norman, 2013; Stanton et al., 2017; Vaiana & McGlynn, 2002). An important element of innovation design may be how the tool or technology is coordinated with other organizational resources and activities (Malone & Crowston, 1994), such as how using a new electronic health record system to help therapists organize information collected during assessment can then be shared easily during supervision.

The results of this study suggest a need for continued efforts from the field to understand how individuals interact with mental health technologies. While our study compared innovation characteristics (conceptualized as an individual’s beliefs about an innovation’s physical features produced by their interaction with the innovation) with individual and organizational characteristics, we encourage the field to adopt an ecological approach in future studies. Beliefs about a specific innovation may vary between individuals, and by extension, a fixed innovation may not be equally suited to all types of individuals, cases, tasks, or organizational contexts (e.g., Buckingham et al., 2019). Said another way, an individual’s interaction with an innovation may capture the relative fit or misfit between an innovation’s physical features, a given therapeutic task, a client’s particular characteristics or preferences, prescribed organizational mandates, and the mental health professional’s experiences and competencies. To facilitate better designs for various mental health contexts, ideally all these factors would be considered in future studies that explore variability in individual beliefs about a specific innovation. The present study was limited in that we could not examine all sources of variability in individuals’ interactions with their innovation. Nonetheless, we hope it highlights the importance of considering these interactions as a starting point for designing interventions for the “user” in context.

Conclusion

For over 60 years, psychological research has studied and incorporated “user” perceptions into technology design (Barki & Hartwwick, 1994; Gerlach and Kuo, 1991; Hackos and Redish, 1998; Proctor et al., 2021), but this knowledge has not been fully utilized within mental health services, despite calls to do so over the past two decades (e.g., Chorpita and Daleiden, 2004; Chorpita et al., 2014). Although there is now a mature and robust literature on human-technology interaction and design (Fowler & Scott, 1997), when developing technologies, the user interaction or experience continues to be a low-priority or late consideration in a process that typically prioritizes function (Burns & Madey, 2001; Randolph, 2004). However, when an engineer prioritizes their technology’s functions above the various needs, goals, and constraints of a multitude of possible users, the result is often a design that is suboptimal to end users, demanding human flexibility and adaptability to make up for any shortfall in the technology design (Mayhew, 1992). Thus, the user pays the price in terms of frustration, inefficiency, and potentially even discontinued use of the technology meant to help them. Examples in everyday life might include remote controls or “smart” technologies (e.g., thermostats, lighting) that have many functions and features but consequently are often frustratingly difficult to use for basic high-priority tasks (Randolph, 2004).

Despite the wealth of information about how individuals interact with technology, this knowledge is not widely considered in mental health treatment design, where the technology is often a treatment manual and the interface is often in a highly detailed narrative book form, with predefined content and sequencing, usually specific to one focal clinical condition. The findings from the present study are in line with roughly six decades of research on user-technology interaction, suggesting that in the mental health context, we might accelerate the uptake and impact of the evidence base by focusing even more earnestly on the fundamental architecture of our best mental health technologies, whose successful designs for the research context might not yet be suitable for those users working in diverse and dynamic service contexts. We challenge the field to go beyond its historical efforts to adapt the thoughts and behaviors of mental health providers or the structure of organizations to “fit” the innovation and begin to rethink its core product designs. Such efforts are already underway (e.g., Becker et al., 2019; Chorpita & Daleiden, 2014; Lyon et al., 2020), and we remain optimistic about the field’s progress towards uniting historically siloed areas of research in the service of youths and families.