Introduction

Eleven million Americans experience psychosis during their lifetime, and roughly 60 million have a loved one affected by psychosis (Palumbo et al., 2015). Family interventions for psychosis (FIp) are intended to enhance the knowledge and skills needed to support illness management in the community (e.g., McFarlane, 2016; Mueser & Gingerich, 2006). A substantial evidence base supports the administration of FIp to enhance functioning, quality of life, and treatment engagement among individuals with psychosis; reducing the risk and duration of hospital admissions (Bird et al., 2010; Onwumere et al., 2011; Pfammatter et al., 2006; Pilling et al., 2002) and minimizing detriments to economic, social, physical, and mental health among family caregivers (Hegde et al., 2019; Stanley et al., 2017). As a result, FIp is widely recognized as the recommended standard of care in both the US and abroad (Keepers et al., 2020; NICE, 2014); yet, an audit of FIp services in the United States (US) suggests that fewer than 2% of family members supporting a loved one with a serious mental illness are receiving any form of FIp (ISMICC, 2017).

To remediate the pervasive inaccessibility of recommended interventions like FIp, federal agencies in the US have advocated for technological strategies to enhance access to quality care (ISMICC, 2017; National Academies of Sciences, Engineering, and Medicine, 2016). Technologic advances in internet-delivered conversational agents and Artificial Intelligence (AI) have broadly influenced the delivery of evidence-based interventions for other health conditions (Vaidyam et al., 2019), and an array of technology-based interventions are specifically geared toward the needs of unpaid caregivers (Czaja et al., 2016). Chief among those needs is on-demand access to accurate information about their loved one’s diagnosis and training in evidence-based skills to improve communication and illness management (Buck et al., 2021). Psychosis REACH (Recovery by Enabling Adult Carers at Home) is an empirically-supported bichronous family intervention that is based on cognitive behavioral therapy for psychosis, an evidence-based psychological treatment for distressing and impairing psychotic symptoms that has amassed a substantial evidence-base and that is included in schizophrenia treatment guidelines (SAMHSA, 2021). Psychosis REACH leverages virtual platforms to provide asynchronous didactics and synchronous skills training to self-identified caregivers in high-yield cognitive and behavioral techniques (CBT) for coping, communication, and problem-solving (Kopelovich et al., 2021; Turkington et al., 2018). Families can access Psychosis REACH independent of a clinical service, thereby redressing the critical access gap observed both domestically and abroad. Despite the fact that Psychosis REACH has enhanced access to much-needed psychoeducation and skills coaching, caregivers have few opportunities to receive feedback on skill rehearsal, which is considered a key element of skill acquisition (Lindhiem et al., 2014) and generalization (Greene, 2003).

Conversational agents offer ubiquitous hybrid (human- and conversational agent-supported) coaching in the caregiving environment, building caregivers’ capacity to enact skills through state-dependent learning and enhancing the alliance between caregivers and their loved ones (Joerin et al., 2019; Lederman et al., 2019). A digital platform with conversational agents represents a low-cost and user-friendly means of increasing health literacy (Ben-Zeev et al., 2021), supporting the delivery of home-based caregiving activities, enhancing communication, and improving caregiving outcomes (Kowatsch et al., 2021; Lindeman et al., 2020). A positive evaluation for integrating conversational agents into family interventions has been demonstrated for chronic disease (Kowatsch et al., 2021). In mental health care (Abd-Alraza et al., 2021), conversational agent-enabled interventions have targeted supporting mental wellbeing for non-help-seeking individuals (Martinengo et al., 2022), assisting behavioral health clients in managing their symptoms (Fulmer et al., 2018; So et al., 2020), and supporting the delivery of psychosocial interventions such as problem-solving therapies or communications skills training in older adults (Ali et al., 2021). There is a lack of digital platforms that deliver evidence-based family intervention to support family caregivers of individuals impacted by serious mental illnesses. To this end, we endeavored to engage end users in the co-production of an internet platform designed to reinforce the evidence-based Psychosis REACH didactic content and foster CBT-informed training among families navigating life with psychosis.

Methods

We identified design requirements based on both the established health technology design framework (Kowatsch et al., 2019) and user interviews before progressing to the participatory, iterative, user-centered design (UCD), and development process. During UCD, we employed individual interviews, group co-design sessions, prototype development (Sefelin et al., 2003) and preliminary usability testing (Cardello, 2014; Nielsen, 2012) of a digitally-delivered CBT for a psychosis (CBTp)-informed skills training platform for family and other caregivers of individuals impacted by psychosis. Compared to other approaches, such as the solution-first approach or function-centered design, UCD systematically identifies user pain points, needs, and goals. As a result, UCD enables co-production between stakeholders (e.g., digital project designers, engineers, treatment developers or content experts, and end users) to optimize the responsivity of the product to the end users’ needs, desires, and contexts (Lyon et al., 2019). Furthermore, involving the intended beneficiaries of interventions in research and the co-production of health services can further enhance the applicability and appropriateness of the intervention (Buckley et al., 2007; Lwembe et al., 2016).

Participants

Our multidisciplinary research and development team worked with an advisory board consisting of family and caregivers (N = 7) of individuals with psychotic and mood disorders. We solicited input on proposal development, design principles, and priority content. We employed purposive sampling to recruit family caregivers who had prior exposure to the Psychosis REACH training to serve as end users. Eligible participants included individuals over the age of 18 who identified as the family or significant other of an individual diagnosed with a psychotic disorder or mood disorder with psychotic features, had previously completed Psychosis REACH training, and endorsed at least weekly contact in any format with a loved one who experiences psychosis. The < BLIND FOR REVIEW > Institutional Review Board approved the study.

Qualitative Interviews

In line with best practices (Patton, 2002), the research team aggregated stakeholder input on relatable caregiving communication challenges, personas, design specifications of the platform, and design requirements. The research team developed a list of semi-structured interview questions. During the individual interviews with the family caregivers, an experienced qualitative researcher used the semi-structured interview guide to elicit family caregivers’ needs for a digital platform that incorporates conversational agents. Interview questions focused on family caregivers’ needs and experiences (i.e., Can you share with me a recent time when you interacted with your loved one while they were experiencing psychotic symptoms? What challenges did you experience with your loved one? Who or where do you go to when you want to know how to communicate with your loved one?) and their interest in integrating technology to support their caregiving (i.e., If we were to put the materials on Psychosis REACH on a website to give you easier access, how would you want it to look/work?). Participants received e-gift cards after the interviews as incentives for their time and involvement. The interviews were audio recorded with permission and subsequently transcribed. Thematic analysis was adopted to analyze the transcripts (Miles & Huberman, 1984).

Co-Design Session

Participants were invited and consented to join the 90-min co-design sessions with the team of UCD experts. The purpose of the co-design sessions was to collectively ideate design specifications for the digital platform that delivers CBTp and has conversational agents to practice the skills with the family caregivers. Participants were first introduced to technology-assisted interventions and provided background information on the proposed digital platform with conversational agents. Participants were asked to brainstorm platform specifications such as intervention modalities, design preferences, user flows, platform layout options, and chatbot conversation format. Based on these design specifications, the research team generated a prototype called psychosis iREACH, a digital platform that focuses on reinforcing Psychosis REACH didactic content and skill rehearsal.

Content Development

Researchers with expertise in intervention, distance learning, and family interventions for mental health conditions collaboratively developed a repository of simulated prompts and interactive scenarios between a caregiver and a loved one with psychosis. The Psychosis REACH treatment development team (second author [blinded] and last author [blinded]) proposed module content and authentic dialogs that conversational agents could use based on their cumulative 55 years of experience working with individuals experiencing psychosis and their families. Scenarios were intended to be generalizable across symptom profiles, cultures, and geographic regions. The research team reviewed and provided feedback on proposed features based on the prescribed parameters of the platform. Two 1-h virtual advisory board meetings were conducted with the goal of soliciting input from the Family and Caregiver Advisory Board. At each meeting, the research team described the requisite components of each training module and the chatbot’s purpose and function.

Usability Testing

The same participants who participated in the previous stages of the study were recruited to provide feedback on the prototype. During the 60-min usability testing session via Zoom, participants were shown the demonstration of the prototype. During the demonstration, participants were encouraged to think aloud (Jaspers et al., 2004) as they interacted with the prototype. After their interactions, participants were asked a series of open-ended questions about their experience navigating the platform (i.e., the homepage, sign-in, and module learning), the interactivity of the conversational agents, and the overall information design. Participants completed the 10-item System Usability Scale (SUS) to assess the perceived level of ease of use and usefulness of the platform. The SUS is a valid and reliable measure that has been widely adopted by studies focusing on the family caregiver population (Quinn et al., 2019). Items are rated on a 5-point Likert scale (1 represents strongly disagree and 5 represents strongly agree), and the total score was derived from the steps suggested by Smyk (2020). According to the developers of the scale, scores exceeding 68 out of 100 indicate above-average usability.

Results

Qualitative Interviews

Of the seven participants, 67% identified as female and 33% identified as male. Most (83%) identified as White, 16% as Native American, and 16% as Asian/Asian American. Most (67%) were a parent of an individual experiencing psychosis, 16% were a spouse, and 16% were a sibling. Regarding technological literacy, almost half the participants (N = 3) reported that they were unfamiliar with trending technology. Forty-three percent of participants reported device use per day ranging from 2 to 5 h, 43% reported 5 to 10 h, and 14% reported more than 10 h. Thematic analysis (inter-coder reliability 89%) revealed participants’ caregiving experiences and their interests in incorporating technology to support themselves. Participants commonly reported their struggles and emotional distress around caregiving. Frustration, exhaustion, and anxiety were common feelings due to their caregiving roles and their worries about the impacted individuals’ safety. Many reported feelings of helplessness, isolation, and confusion, commonly generated during times when they were unable to access resources. For example, a caregiver stated “I feel I am on my own and no one has the answer to my questions.” Participants endorsed a preference for an on-demand web-based tool with little-to-no prerequisite digital skills. One participant noted, “To have to learn new things all the time…for someone who has a child, with an adult child with psychosis, that’s stressful enough. [I’d] just rather have things be really easy, so no password and one way to do everything and a clear explanation of how to use the site in writing.” Participants also expressed the need for a community that knows how to navigate caregiving challenges so they can learn from others’ experience. “… we [are] all in so much pain about... our own family member, where we see... our own stories and the beauty of stopping that and listening to [another] person in their own pain. You know that has a transformational power that it takes you away from.” When asked about the main challenge they experienced with their loved ones, the dominant theme across participants was a perceived lack of competence in communicating with their loved one. For example, one participant said, “I would love more answers to know how to approach the patient.” Overall, participants expressed the need to have a tool for effective communication with their loved ones who suffered from psychosis.

When asked about their interests in incorporating a digital platform with conversational agents, users first expressed positive sentiments regarding the Psychosis REACH training. The training has equipped the caregivers with useful techniques to communicate more effectively with their loved ones. “I took the training in May…It was amazing, super informative, very thorough, great tools…strategies like empathizing are great.” Participants suggested that there is a need to review, practice, and receive feedback on Psychosis REACH skills at their own pace. When asked about appropriate formats to communicate with the coach and the simulated patient, participants agreed using multiple choice response options is more preferable than free text typing with responses. Overall, thematic analyses revealed the following three features of the desired digital platform: (1) an online platform in which users can access the Psychosis REACH content whenever they need to know how to approach their loved ones, (2) a conversational agent (from now on, referred to as “coach”) that helps users navigate the platform and provides instructions and feedback to users through multiple choice response options, and (3) a conversational agent that simulates a loved one diagnosed with a psychotic disorder (from now on, referred as “client”) with whom users can practice each of the five Psychosis REACH skills.

Developed within the context of the health technology design framework (Kowatsch et al., 2019), participants’ feedback was distilled into six core design principles, which served as a guideline for the platform design team (Fig. 1). The six design principles are simplicity, usability, accessibility, skill acquisition, sense of community, and hope.

Fig. 1
figure 1

Psychosis iREACH design principles and requirements

Full size image

Co-Design Session

The same seven participants who participated in the qualitative interviews were invited to the co-design sessions; all participants participated. Based on the findings from the qualitative interviews, the design sessions were organized into three parts: the design of the online platform, the coach conversational agent, and the client conversational agent. With regard to the design of the platform, participants recommended that learning modules be presented as optional features for learners to access on demand. Participants noted a preference for these modules to present narrative summaries of key information (e.g., rationale, empirical basis, and core techniques) associated with each skill domain to enhance comprehension and support deeper learning for self-selecting learners. Participants’ feedback on design features also included preferences for the sign-up and login page, duplicating navigation to skill modules on both the side and bottom of the screen and boxing text into dialog boxes to enable easier tracking between the user, coach, and client conversational agents. For the sign-up page, participants expressed that 2-factor authentication seemed secure and trustworthy. They suggested that security statements should be added to explain why the website collects personal data (e.g., email and/or phone number, relationship to loved one with psychosis, and date of birth) and how data is stored. Features such as a magnifying glass function, larger font size, video and audio media, and print functions for each module were requested to enhance accessibility. Regarding the coach conversational agent, participants requested easy access to the virtual coach, supported by a floating button that tracks along with user progress on a screen and also fixed to the top right corner of the screen. Participants also indicated that depicting the coach conversational agent as one of treatment developers gives the coach more credibility. Participants viewed a series of 3 mood board visual prototypes to assess their preferences on the conversation format layouts. Option A is the conversational agent in the middle with the show text option, option B is the conversational agent in the middle with text always present, and option C is the conversational agent at the left side. One participant (14%) chose A; 5 participants (71%) chose B; 1 participant (14%) chose C. Finally, although participants reported a preference for representation of diverse clinical presentations and relationships between the caregiver and loved ones with psychosis, stakeholders also reasoned that the initial prototype should focus on one common clinical symptom and one caregiving relationship. Because parents are the most frequent caregiver to access Psychosis REACH trainings (Kopelovich et al., 2022, 2023), the determination was made to prioritize the depiction of a parent interacting with an adult child experiencing persistent paranoid ideation.

Content Development

Modules were developed to correspond to each of the five categories of cognitive behavioral skills taught in Psychosis REACH, which are abbreviated using the acronym FIRST (Kopelovich et al., 2022). Each of the five modules includes a specific learning objective, a brief summary and rationale of the skill domain, and real-world examples of skill applications. Following the didactic content, users are instructed to self-navigate to the client conversational agent to practice each set of skills.

Usability Testing

The same seven participants who participated in the co-design session were recruited for usability activities to verify the alignment of changes with initial feedback. Regarding the website platform, the overall feedback was that users had no barriers in navigating the website, which they noted was simple and straightforward: “I like that the sidebar with all [menu] choices continues onto this page so anytime I can go back to any menus on the sidebar menu. The website is very simple.” Changes were made to avoid cognitive overload with too much text-based visual stimuli: “Too much text makes it hard to concentrate.” To improve users’ experience, we increased the font size to 18 points, used visually appealing blocks to chunk content, added printer tabs, and augmented auditory and visual support (e.g., adding a magnifying glass function). Additionally, we added a progress indicator for users to know how much progress they have made in using the website to learn FIp and how many rounds of dialogs they have had with the client conversational agent. Participants expressed their appreciation of having a progress bar, stating “I like the percentage present. I would know the progress and it gives more information.” For both the coach and client conversational agents, participants expressed a preference for personalized avatars. For example, if the coach’s conversational agent depicts the experts involved in the Psychosis REACH training, it enhances familiarity and credibility. One participant mentioned, “I like the real image and makes it feel like you have someone actually there. It’s not silly and more personalized.” Another participant stated, “I prefer the image of someone from the team.” Interactivity is important for our participants (“I think the animated avatars are more engaging than text only.”).

Usability interviews were designed to understand the levels of ease of use and usefulness of the website. Reported levels of use of use ranged from 4.2 to 4.8 (1 being very difficult and 5 being very easy); levels of usefulness ranged from 4.0 to 4.7 (1 being very useless and 5 being very useful). User feedback on the website difficulty levels included: “The design does not [look] like I am reading a book or taking a class. It is like conversations… everything is so good. I want to continue (browsing the website)” and “Everything is easy to read, especially when we are always in a hurry.” Scores on the SUS indicate that participants generally feel that the Psychosis iREACH user interface is easy to use and found that the coach and client conversational agents feature was highly intuitive. The SUS mean score 93.1 (SD = 7.78) indicates above-average usability.

Discussion

Education and skills training related to symptom identification, monitoring, communication, and illness management have been consistently recognized as a significant challenge for caregivers of individuals with psychosis (Cheng et al., 2020; Kopelovich et al., 2021). Psychosis REACH is intended to enhance access to caregiver training in these evidence-based skills for psychosis, yielding promising effects on the wellbeing and efficacy of caregivers and potentially to the individual experiencing psychosis (Kopelovich et al., 2023; Rathod et al., 2023). While Psychosis REACH’s bichronous training approach can enhance the reach of the intervention (e.g., Kopelovich et al., Under Review), a more dynamic platform is needed to enhance opportunities for learners to rehearse the skills. The conceptual impetus for Psychosis iREACH was to leverage conversational agents to create dynamic opportunities for Psychosis REACH trainees to reinforce didactic content and enhance skill acquisition. We adhered to best practices in digital platform development to create a Psychosis iREACH functional prototype. While limited by a small scope and sample, our co-production approach can be extended to caregivers of individuals with other serious mental health conditions. For example, the real-time collaborative prototyping efforts using remote teleconference sessions was effective, feasible, and acceptable among family caregivers of individuals with psychosis. The process resulted in a functional prototype that family caregivers found to be accessible, acceptable, and usable. These promising findings set the stage for further development and testing.

Our research team anchored accessibility and usability in each stage of the design process. We engaged end users with various technical skills in their home or work environments to ensure that the product was navigable for diverse users and devices. According to our research, conversational agents can be embodied with animated visual representations of field experts so users feel more engaged. Several national guidelines have recommended that both CBT and FIp be made more widely available (National Institute for Clinical Excellence, 2002). Using digital platforms to deliver CBT for psychosis to family caregivers increases accessibility. A systematic review (Kambeitz-Ilankovic et al., 2022) also indicates comparable effectiveness of face-to-face and digital CBT approaches. In our study, participants reported the interests and usability of the digital platform, specifically, the conversational agents, to practice and receive real-time feedback. Literature demonstrates that appropriately designed conversational agents utilize natural language processing (NLP) techniques to support learning (Chhibber & Law, 2019). The benefits of using conversational agents to assist learning include fostering an environment with ease and comfort to both fulfill the need for interpersonal relationships in learning and stimulate essential learning behaviors through just-in-time information exchange and communication (Figen & Ozlem, 2012). Our preliminary work has led to the development of an appropriate prototype. The natural next steps are to use the platform to collect more language data for optimizing NLP techniques and family caregivers’ learning experiences.

In the usability stage, participants rated both the platform and the conversational agents feature as highly effective, efficient, and satisfactory. Users reported that the design is simple and easy because of the consistency of the design and clear descriptions of each function and stated they would recommend it to others. Several preferred features from the usability interviews included privacy statements on the sign-up page, tooltips in each menu option, and the skip options for returned users, which can be extended to other technology-assisted family intervention platforms.

Limitations and Implications

UCD research is often limited to small samples (Greer et al., 2019). Our sample, while small, was enriched by direct stakeholder involvement in the discovery and initial design process. That said, digital training tools can and likely will be deployed internationally and should therefore represent the needs and desires of the target audience in diverse geographic regions. Future research and development should therefore focus on bolstering the number and diversity of stakeholders involved in the co-production process and enhancing the representation of various clinical presentations and caregiving relationships.

This study is aimed at designing and testing Psychosis iREACH, a digital platform consisting of web-based modules and functional conversational agents to enhance knowledge and competency in cognitive behavioral techniques for families caring for a loved one with psychosis. The study benefited from the inclusion of participants who had lived experience as caregivers of individuals with psychotic disorders, had completed the Psychosis REACH core training, and who reported a range of comfort levels with technology. Future platform development can be enhanced by aggregating suggestions from a larger and more diverse sample of caregivers to enhance interactivity and expedient access to the platform.