Introduction

Assertive Community Treatment (ACT) is a well-established, evidence-based treatment model for people with serious mental illnesses (SMI) in a community setting (Bond et al., 2001; Stein & Test, 1980). As part of the community mental health services, ACT is highly effective in reducing illness burden, and is often described as the “gold standard” of treatment in community psychiatry (Dixon, 2000). One of its key innovations as a model is delivering treatment and care through outreach and active engagement in the community. Social distancing and minimizing personal contact as mandated by the Covid-19 pandemic has had fundamental impact on how ACT services were delivered. Few specific guidelines existed to guide ACT teams in responding to disruptions at this large scale (Druss, 2020), putting the already highly vulnerable SMI populations at further risk (Kozloff et al., 2020).

Although there are some general and early reports of the pandemic’s impact on ACT and community psychiatry (Couser et al., 2021; Guan et al., 2021; Neda et al., 2022), relatively little is known regarding how and to what extent the quality of community treatment models like ACT have been affected. ACT utilizes a highly coordinated multi-disciplinary team, involving psychiatrists, nurses, social workers, occupational therapists, addictions workers, peer support workers, and others. Regular daily team meetings, after-hours on call system, and serving patients in their milieu, using outreach, community and home visits, and providing intensive psychosocial services are at the core of its service model (Bond et al., 2001). There are various aspects of this complex service model that benefit from standardization, which in turn helps to gauge its service quality and level of functioning. The health of the ACT model is also a reflection of the quality of the community psychiatry sector as a whole. Historically, ACT has utilized model fidelity instruments such as the Dartmouth ACT Scale (DACTS), which outlines in detail the structures and functions of this service model, and sets the standard (Teague et al., 1998). Research also shows good correlation between high program fidelity and patient outcomes (Bond & Salyers, 2004). DACTS has three main domains: Human Resources (i.e., caseload ratios, clinician composition, turnover rates, etc.), Organizational Boundaries (i.e. intake and discharge criteria, scope and division of clinical responsibilities, responsibility for crisis services, 24-hour on call system, etc.), and Nature of Services provided (i.e. engagement approaches, medication adherence monitoring, and peer support) (George et al., 2009). Studying these domains provides a good opportunity to understand the impact of the pandemic on community psychiatry and operations of ACT in a comprehensive, standardized and quantitative way.

Since 1998, the province of Ontario, Canada, with a population of 14.8 million, has developed over 80 ACT teams, forming the core of the most intensive services for people with SMI (George et al., 2009). Most Ontario ACT teams were considered mature, stable, and with good fidelity (George et al., 2010). During the pandemic, Ontario has registered 1,616,240 Covid-19 cases and 16,488 deaths as of April 29, 2023 (Public Health Ontario, 2023). While the current pandemic is easing to a substantial degree, future recurrence remains a real challenge (Moore et al., 2020). Using DACTS, we surveyed the ACT teams in Ontario to understand the level of ACT quality before and after the pandemic. We aimed to learn the impact of the pandemic on ACT, and the timely evaluation could inform current quality of the provincial ACT teams, pandemic related program adaptations, areas of concern, and help to prepare for future major disruptions.

Methods

The current study on the impact of Covid-19 pandemic on team structures and functioning of Ontario ACT teams is part of a larger study that examined Covid-19 related adaptations and innovations in community psychiatry in Ontario. The research team developed the current ACT fidelity survey modeled on a previous study by George and colleagues in 2007–2008, who surveyed ACT team clinical and administrative leaders in Ontario using DACTS through a self-report approach (George et al., 2010). While the standard of fidelity evaluation using DACTS is done on-site by trained assessors, research shows self-reported DACTS assessment could produce reasonably reliable results (McGrew et al., 2013). Other successful self-reported fidelity surveys are found in measuring Individual Placement and Support Program (IPS) (Margolies et al., 2018), and Dual Diagnosis Capability in Mental Health Treatment (DDCMHT) (Covell et al., 2021). While non-ideal, experts also see a role for self-reported DACTS in some circumstances and for quality improvement purposes (Bond, 2013). Beyond these, the current study’s self-reporting approach is also related to the restrictions placed on time, budget, and in-person contact related to the pandemic. The original published version of DACTS was used for the survey using an online platform hosted by SimpleSurvey (SimpleSurvey.com).

The DACTS is a 28-item measurement, where each item is rated on a 5-point scale, with 1 being not implemented to 5 being fully implemented (Bond & Salyers, 2004). The 28 items are organized into three domains: Human Resources (11 items), Organizational Boundaries (7 Items), and Nature of Services (10 items) (Bond & Salyers, 2004). We modified 3 relevant questions (i.e. Human resources, question 2; Nature of services questions 4 & 5) to include “virtual” care in clinical contact time to capture adaptations made during the pandemic. (See Table 1 for details of the 28 items and the 3 domains of DACTS). The survey took typically 20–40 min to complete.

The study targeted the clinical and administrative leaders of the 80 ACT teams in Ontario, who belong to the Ontario Association of ACT and FACT (Flexible ACT) (OAAF, 2022), the sole provincial organization that engages in standard and quality improvement, professional education, and political advocacy. The OAAF maintains an active mailing list that reaches 232 individuals, who are made up of 2 to 4 members from each of the 80 ACT teams in Ontario. The mailing list members are typically team leaders, team managers, or senior members. Each potential participant was sent a copy of the study information and consent form, and a link to the SimpleSurvey on April 12, 2021. Participants were those who have worked on the team for at least 3 or more years and were asked to use their best recall and judgment, and any relevant objective information available to rate the fidelity of the team during the one-year period before the pandemic (i.e. 2018–2019), and the most recent one-year period post-pandemic (i.e. 2020–2021). Four reminder emails followed over the course of five weeks. Response collection closed on May 31, 2021; the surveyed period included part of the height of the Delta variant “third wave” Covid-19 pandemic (Public Health Ontario, 2023). All questions on the survey were optional, and responses were collected anonymously. To encourage fuller disclosure, except for one question that asked for the geographical type (e.g. small, medium or large population centres) in which the respondent’s ACT team was located, no information was collected that would link the respondent’s response to the name of the team, or clinical position of the respondent on the team. Only completed and submitted responses are included in this analysis. We use descriptive information and 2-sample t-tests statistical analysis for the comparisons of before- and after-Covid DACTS scores.

Results

The final data set consisted of 144 completed surveys for an estimated completion rate of 62.1%. We received 32 responses from Metro Toronto (population 5.4 million), 57 from other large centres (population > 100,000), 29 from medium (population 30,000–99,999), 20 from small (population < 30,000), 4 from rural regions, and 2 unknowns. The proportions of responses from each geographical region matched well with the actual distribution and proportion the ACT teams in those geographical centres.

As outlined in Table 1, our study shows the overall mean DACTS score - by combing all participants’ answers across Ontario - from one-year pre-Covid (2018–2019) was 3.65 (Human resources 3.66, Organization boundary 3.89, Nature of Services 3.40). This score corresponds to a medium fidelity by DACTS standards (between 3.0 and 3.9). The item-specific fidelity scores show low fidelities in attendance in substance abuse treatment (1.39), and team’s approach in addictions (2.91). The high fidelity areas were numerous - some highlights include staff to client ratio (4.34), team approach in patient contacts (4.19), low turnover (4.24), having full nursing staff (4.41), regular new intake (4.84), full range of services (4.21), appropriate discharges (4.57), community milieu contacts (4.42), case retention (4.51) and active engagement (4.74).

When compared to the year pre-Covid – using 2-sample t-tests, the overall post-Covid DACTS scores declined by 3.84% (p < 0.001, 95%CI [0.09–0.19]) across the province. The subcategory of DACTS fidelity scores all declined during the pandemic: Human Resources (HR) dropped by 4.92% (p < 0.001, 95%CI [0.078–0.27]), Organization Boundaries (OB) by 1.03% (p < 0.013, 95%CI [0.01–0.07]), and Nature of Services (NoS) by 6.18% (p < 0.001, 95%CI [0.16–0.26]) (see Table 1 for details). The regional differences were notable: regarding HR: Small centres and Rural area combined declined (−0.31%, p < 0.97, 95%CI [−0.37–0.39]), and Metro Toronto (−3.46%, p < 0.29, 95%CI [−0.12–0.39]) had the least impact, while medium (−6.06%, p < 0.025, 95%CI [0.03–0.41]) and large centres (−6.49%, p < 0.001, 95%CI [0.17–0.32]) were more affected. For OB, the impact was minimal, with all regional changes being under 2% and only metro Toronto had a significant decline (−1.20%, p < 0.017, 95%CI [0.008–0.078]). For NoS, the magnitude of change in this domain was the largest overall, with all regions experiencing significant changes between 5 and 7%, with the highest being in large centres.

More specifically, the most prominent DACTS changes were under HR, where the 3 most significant changes were (1) reduced face-to-face interactions with more than one clinician (-27.92%, p < 0.001, 95%CI [0.94–1.40]), (2) more staff turnover (−2.36%, p < 0.0088, 95%CI [0.024–0.16]), and (3) less full complement of staff (−2.36%, p < 0.001, 95%CI [0.063–0.21]).

For OB, there were also 3 significant changes: (1) reduced admission (−1.25%, p < 0.014, 95%CI [0.0089–0.077]), (2) reduced services (−1.19%, p < 0.019, 95%CI [0.0083–0.092]), and (3) less involvement in client hospital discharges (−2.62%, p < 0.033, 95%CI [0.0068–0.16]).

For NoS, where most of the impact occurred, there were 6 significant findings. These included (1) decreased community visits (−9.73%, p < 0.001, 95%CI [0.18–0.68]), (2) less time spent in interactions (−17.50%, p < 0.001, 95%CI [0.47–0.79]), (3) lowered face-to-face contacts (−17.82%, p < 0.001, 95%CI: [0.40–0.68]), (4) decreased contacts with client’s support network (−4.29%, p < 0.050, 95%CI [0.00032–0.29]), (5) less services for addictions (−1.93%, p < 0.011, 95%CI [0.013–0.10]), and (6) less attendance of addictions treatment groups (−14.39%, p < 0.0043, 95%CI [0.064–0.34])

Overall, out of the 28 DACTS items, 12 items were affected, while the remainder remained relatively unchanged and intact. Highlights of the unaffected areas (all under 3% change, and all p > 0.05) were staff to client ratio, team meetings, availability of psychiatrist, nurses, addictions specialists, vocational specialists, and full-time staff, after-hours crisis response, hospital admissions assistance, client retainment, team approach in substance misuse issues, and peer support workers providing direct services.

(Of note: Compared to the similar survey conducted in 2008 (George et al., 2010), where the overall fidelity was 4.22 (considered a high fidelity score; specifically: Human resources 4.25, Organizational boundaries 4.61, Nature of services 3.92), the current pre-Covid score of 3.65 showed an overall decrease of 13.5%.)

Table 1 Evaluation of each of the 28-item DACTS for all ACT teams across Ontario, Canada
Full size table

Discussion

Using the well-researched DACTS, the current quantitative study shows a number of significant specific changes in ACT services in Ontario, Canada during the pandemic. These involved, not surprisingly given the limited person-to-person contact by public health mandates, reduction in number of face-to-face time with clients, community visits, and psychosocial programs and support, including access to substance use groups, ACT being less involved in discharge planning, and reduction in clinician and psychiatrist staff. On the positive side, the results also show numerous higher fidelity fields pre-Covid in areas such as caseload, overall staff stability, range of services and outreach engagement, among others. However, there are notable areas of low fidelity such as a lack in addictions treatment, vocational specialists, and low after-hour care, making the overall fidelity wanting even before the pandemic, with substantial drop from a decade ago.

Regarding pandemic related changes to team quality, the most affected, as expected, were related to the Nature of Services domain, with loss of face-to-face contacts and appointments, time spent, community visits, and psychosocial and addictions support. To compensate, studies have shown that many ACT services resorted to alternative ways such as virtual care, using different (e.g. longer acting injection) medications, collaborations with allied professionals and families, etc. during the pandemic (Couser et al., 2021; Guan et al., 2021). These adaptations were not likely captured by the DACTS. However, incorporating the positive adaptations and preserving the strengths of person-to person contact should be a priority in post-pandemic recovery for ACT services. Research has consistently shown that community outreach is a key ingredient of ACT effectiveness (Bond & Drake, 2015), and most clinicians value the depth and quality of in-person care (McGrew et al., 2003). Regarding virtual care, while there are some benefits to such, its quality and desirability are still uncertain, and client access to technology is also a barrier (Tse et al., 2021; Zhou et al., 2020). Further research to optimize a hybrid model that includes in-person and telehealth options is warranted (Rosenheck et al., 2021; Zulfic et al., 2020).

The findings of higher staff turnover and lowered availability of workforce are of note. Whether these changes were temporary, reflecting the known disruptions from infectious illness, and staff burnout, or something longer lasting, is a potential area of concern. ACT services rely on a sub-specialized and dedicated work force to offer a full range of services. Such threat of attrition or staff shortage may differentially affect provisions such as peer support, employment, and housing services, as reported by a recent pre-pandemic survey (Spivak et al., 2019). The current study may highlight where future services planning need to monitor.

The current study also found regional differences in Ontario. Of note, the impact on rural areas and small centres were generally smaller when compared to those of the larger centres. This was possibly reflecting the pandemic’s lower impact in less densely populated areas, allowing the core of ACT services to be relatively preserved, despite their less resourced pre-existing condition that are typical of rural North American ACT teams (George et al., 2010; Meyer & Morrissey, 2007; Stefancic et al., 2013).

In terms of Organizational Boundaries, the changes were relatively minor. However, as expected, the rates of admission to ACT were affected. Responsibility for client care was also affected as the front-line, in-home services happened less frequently. Some of these impacts may have been felt by allied professionals as there are reports that suggest Emergency Room, ambulance services, or even police were more likely to be involved in servicing people with SMI during the pandemic (Laufs & Waseem, 2020; Tuczyńska et al., 2021). Being less involved for clients’ hospital discharge was also a significant finding. While this may be related to pandemic hospital protocol changes, close monitoring of this key ACT service and function should be done to ensure high quality, seamless services at a highly vulnerable time for clients (Cutcliffe et al., 2012; Goldacre et al., 1993).

Overall, despite a very disruptive crisis at a global level, the magnitude of changes to the ACT services as monitored by DACTS was relatively modest – mostly under 3%, with pandemic related exceptions. These are modest changes when compared to those areas of medicine that have experienced more extreme disruptions, including primary care, general psychiatry, orthopedics, cardiology, neurosurgery, and others according to a recent review (Tuczyńska et al., 2021). Relatedly, the ACT model may have advantages in its design and philosophical approach that emphasizes strong professional dedication, quick decision-making capacity, positive problem-solving attitudes, task and burden sharing, a relative horizontal organizational hierarchical structure, and mutual support (Bond et al., 2001). These attributes may ultimately support resilience and adaptability in the service model at a time of a major disruption (Bommersbach et al., 2021; Dixon, 2000). In other words, ACT is set up and trained to manage crises, including the likes of a global pandemic. The relative modest change may also be related to the fact that ACT is created with a robust range of resources and scope of services, allowing it to be flexible and adaptable (Bond & Salyers, 2004; Schöttle et al., 2014). The key ingredients are also wide ranging, with multiple, likely additive and interactive ways to address the complex needs of people with SMI (Bond & Drake, 2015).

Last but not least, a surprising and important finding was that at a larger picture level, our study shows the overall one-year pre-Covid (2018–2019) fidelity of ACT teams in Ontario were on average of medium fidelity (3.0-3.9). Compared to the previous similar survey in 2007–2008 (George et al., 2010), which found Ontario ACT with an overall fidelity score of 4.22 (high fidelity), the current results show a substantial decline in the last decade, by about 13.5%, a drop much higher than that related to the pandemic. Some changes in fidelity scores over 13 years are expected, but given that the two surveys used the same DACTS instrument and very similar survey methods, the consistent downward trend and large differences are worth pondering. The areas that showed the lower fidelity were in addition services, vocational support, and after-hours care, among others – all related to some of the areas that ACT needed to do to increase high quality wrap-around care. Reasons for such an insidious drop may be complex, likely related to a relative global funding decline in the community psychiatry sector in Ontario, and or a lack of commensurate increase in resources as client populations and demand for services increased (Bartram, 2017). This disquiet about a lack of resources is not unique, as it echoes concerns in other North American jurisdictions (Moser et al., 2004; Spivak et al., 2019). The differences between the current study and the 2007–2008 one could also be due to slight differences in methodologies - the earlier survey was recorded in real time, and the current study relied on recall about a period 2 years ago, the overall magnitude of deterioration was large and unmistakable, and attention must be paid to address this downward drift in quality of ACT teams in Ontario. This is particularly salient as the pandemic related negative changes are more easily ameliorated, as in-person contacts and time spent with patients are rebounding when the pandemic recedes. The historical changes in the last 13 years in terms of loss of addictions and vocational support would require a closer examination and may require system level intervention.

While having in-house capacity to address addictions and vocational challenge are likely part of the key ingredients of ACT’s success (McGrew & Bond, 1995), one possible explanations of the current study findings of their loss from ACT teams could be that these services have been “out-sourced” to other community providers. From a front-line community service providers’ perspective, we have witnessed development of more addictions services such as detoxification beds, assessments and counselling resources, and opioid replacement therapy, etc., in some regions of the province. How well these are integrated into the community mental health system, and how much of these services are accessed by ACT clients are not well known, and deserve additional research. What research does show is a large up-tick of addictions problems, particularly those related to opioid and stimulant use in the province at large, overwhelming existing services (Gomes et al., 2023; Kourgiantakis et al., 2023). Other challenges include a shortage of residential treatment, gaps in program types, barriers to access, and regional imbalance in services (Government of Canada, 2018; Mandal & Burella, 2021; Rush et al., 2021). Reliance on Emergency Room for addictions treatment is common (Matsumoto et al., 2017). These system and social level realities make the lack of addictions services on ACT more troubling overall. The picture on vocational support is likely worse, as little to no employment support programs for people with mental disabilities have been developed in the province at large (Rebeiro Gruhl, 2012; Latimer et al., 2020; Menear et al., 2011). In summary, the decline in both addictions and vocational support on ACT teams likely have direct negative impact on the quality of ACT services.

Lastly, little research is available to inform the field how some of the pandemic related adaptations (e.g. Guan et al., 2021; Couser et al., 2021; Law et al., 2021) have impacted on the quality of ACT services. There are some preliminary reports that highlight a mixture of negative clinical impacts, increased addictions issues, emergency service involvement, and positive adaptations and resilience in ACT settings in Ontario and elsewhere (Kassam et al., 2023a, b; Motamedi et al., 2022). Based on the current DACTS survey, the magnitude of negative DACTS changes over the pandemic seem limited. In addition, there may be aspects of positive changes that were not captured by the DACTS instrument. Case in point, researchers have found that the pandemic has had relatively minor impact on the Individual Placement and Support (IPS) programs - another fidelity based service - in New York state and elsewhere as they shifted to virtual or remote form of services (Margolies et al., 2022; Wittlund et al., 2023). Moreover, when researchers adapted the fidelity instrument to accommodate pandemic adaptations such as remote services, the pandemic related changes were minimal (Margolies et al., 2022). In the current study, despite the inclusion of virtual care, DACTS scores still registered a decrease in quality, suggesting the pandemic has had more substantial impact on the out-reach oriented and more complex ACT services.

Limitations of the current study include the use of self-reported DACTS, which may not be as reliable as on-site evaluation by trained assessors (e.g. Lee & Cameron, 2009). This less-than-ideal but real-world adaptation may be better understood through the lens of a quality improvement or monitoring – which DACTS is also known to be suitable (Bond, 2013) - during the crisis of a pandemic. The Ontario teams surveyed are experienced ACT teams that have typically been in existence for more than 20 years, so the DACTS was used less for establishing whether they qualify to be an ACT - as standard DACTS assessment are meant to be, but evaluating how well they function as a ACT team (Salyers et al., 2003). Another limitation is the fact that some ACT teams were over-represented in the current survey that used the average DACTS scores (144 responses for 80 teams). Given the nature of the anonymous survey, we did not track the exact teams from which the respondents came. This could have introduced a bias of the data in either direction. Similarly, inter-rater reliability test was not feasible. The cross-sectional survey for assessors to recall two separate time periods may also introduce recall bias and the direction of bias is unclear from the data. As research shows, a prolonged recall period likely will lead to recall biases and negatively affect data quality (Te Braak et al., 2023). Future studies should aim to reduce the recall periods, encourage the use of memory aids, use standard instruments and objective, validated, and time stamped materials to substantiate the responses, and evaluate the agreements between responses from self-reporting surveys and gold-standard evaluations to check reliability and validity (Biemer et al., 2013; Althubaiti, 2016). Noting our study’s limitation, one would have anticipated some bias in the direction of overestimating the negative impact of the pandemic based on the dread and frustration with the pandemic, and possibly rosier recall bias for the “normal” and better times in the pre-Covid period. However, contrary to these anticipated directions of biases, the study results showed consistency in the trends and a relative small difference between the pre- and-post Covid DACTS scores, making the study results quite remarkable. More interestingly, the bigger difference was the historical drop from the previous provincial study. There is also a lack of qualitative information to contextualize the quantitative findings. The Ontario based setting may limit the representativeness and generalizability of the study. While it may be counter-intuitive to measure fidelity during a time of a major –hopefully temporary - disruption, the current results do provide a quick sentinel survey of the recent past and pandemic related changes, and inform on areas to improve for the future, as well as provide some reassurance on the resilience and stability of the sector and ACT as a service model. Lessons learned from the current study could be useful to prepare for future disruptions, and to sound alarm for the current state of its overall fidelity compared historically. How these changes have already affected client stability, clinical outcome, and recovery remains to be seen.

Implications for Behavioral Health

This timely measure of the impact of the pandemic on the team quality of a major service sector that services people with severe mental illness shows specific loss of services related to public health mandates, which were predictable. There were some areas that were more insidious and would require close monitoring lest the erosion from the pandemic is more permanently set and affect the overall quality of care from this crucial sector. The overall fidelity of Ontario ACT teams were medium pre-pandemic, and showed a decline from similar survey a decade ago. This notable change suggests possible decrease in resource support and the trend is alarming and deserve strong advocacy and prioritizing for a sector that serves one of the most marginalized and vulnerable populations. The overall relatively modest change in team quality suggests ACT model resilience and design advantage in coping with large-scale disruption. Quantitative evaluations to ascertain disruptions and their historical changes and impact are well warranted,