Background

Health research in Canada is moving towards greater engagement with patients, their families and caregivers, and others with an interest in health research decision-making [1]. This shift is evidenced by the increasing prevalence of a patient-oriented approach to research; the Canadian Institutes of Health Research (CIHR) recognizes patient-oriented research as a cornerstone of evidence-informed healthcare [2]. Patient-oriented research involves meaningful and active engagement with patient partners throughout the research process, which can include governance, priority setting, study design and execution, and knowledge translation and mobilization [3]. In contemporary health research parlance, the term “patients” may refer to individuals receiving health care/using health services or to their caregivers (insofar as the latter are involved in facilitating care in a non-professional capacity). In this article, “patients” will refer to health service users, and “patient partners” will refer to a broader group of individuals, i.e., those with lived experience in health care who are partnering with researchers to plan, conduct, implement, and evaluate health research projects. This robust involvement of patients and caregivers in health research is intended to promote increased attention to their priorities and, in turn, improve health outcomes for all Canadians [1, 4,5,6]. To be conducted ethically and effectively, patient-oriented research requires distinctive skills and knowledge on the part of all research team members, making capacity building a key priority for all individuals engaged with health research [7, 8]. The intertwining logistical and ethical complexities of involving patients in health research call for rigorous yet accessible training [8]. To date, capacity for patient-oriented research across Canada has seen significant but uneven progression [9,10,11,12]. However, a growing international trend of governmental support for co-produced health research that emphasizes patient engagement has recently emerged [4]. Mounting interest in and support for patient-oriented research is accelerating the development of more formalized training for research teams.

In 2016, the Canadian Institutes for Health Research created a training curriculum to help guide research teams in applying the principles of patient-oriented research, as part of the Strategy for Patient-Oriented Research [13]. Their Foundations in Patient-Oriented Research, initially created as an in-person course integrating a balance of traditional coursework and peer-to-peer learning activities, was co-developed by a diverse collective of researchers, patients, educators, a healthcare professional, and a health system decision-maker [4]. The curriculum adopts a co-learning and co-facilitating format wherein different stakeholder groups lead and learn together, thereby modeling the collaborative nature of patient-oriented research [4, 14]. Evaluation of the course yielded overall positive feedback, with the curriculum’s co-learning format and interactivity indicated as noteworthy strengths contributing to its success. Recommendations for improvement included incorporating more examples and practical information [14].

Despite this positive impact, however, accessibility barriers to in-person training opportunities such as the Foundations course have persisted. For example, potential learners may confront logistical barriers that inhibit their participation in patient-oriented research capacity building activities [15, 16]. The demand for patient-oriented research training, however, has only continued to grow. Facilitating skilled and impactful involvement of diverse patients and others in health research thus requires innovative approaches to increasing capacity in patient-oriented research [10, 16,17,18]. In response to this need, online courses in patient-oriented research are increasingly available across Canada [19]. Virtual course delivery carries multiple benefits; for instance, it streamlines distributing and updating course content, as well as tailoring the pace and sequence of learning [12, 20,21,22]. These features may be particularly advantageous in the context of patient-oriented research training environments, wherein learners often have different skill sets, learning needs, and backgrounds.

Goals and objectives

To help surmount accessibility barriers in disseminating patient-oriented research training and achieving greater capacity across Canada, our team adapted the Foundations course, developed by the Canadian Institutes of Health Research in 2016, for online delivery. Widely available training opportunities for patients, researchers, trainees, clinicians, and decision-makers will provide the necessary foundation for nation-wide expertise in patient-oriented research, and in turn, for the improved patient outcomes that served as the original motivation for the Strategy for Patient-Oriented Research. To this end, the current evaluation focuses on the following research questions: (1) How is the online delivery of the course Foundations in Patient-Oriented Research received by participants? and (2) What are their recommendations for improving similar training curricula in patient-oriented research training?

Methods

Study design

This online course in patient-oriented research was evaluated using an online post-training survey, which was completed by learners and facilitators following completion of each course module.

Course design and delivery

The original course was adapted for online delivery by the research team, in collaboration with a diverse team that included researchers, an online learning expert, and a patient partner who helped lead the development of the original course. Like its predecessor, the online course provides participants with a comprehensive introduction to patient-oriented research and Canada’s health research landscape, as well as enhanced skills in teamwork. It follows the same three modules as the original course and maintains the co-learning and co-facilitating structure. For this project, each session’s learner group was comprised of research teams that included patient partners, researchers, and other project collaborators. Each session was facilitated by a team that included at least one patient partner. Four course facilitators were also co-authors of this article (LM, AL, LW, SD), and two of those authors (LM and LW) completed the facilitators’ feedback survey.

The first module, “Introduction to Patient-Oriented Research”, is the longest of the three; it includes three synchronous virtual sessions, each about two and a half hours long, and approximately one hour of pre-work for each session. It aims to provide a basic understanding of patient-oriented research and helps participants decide whether and how they might wish to be involved in the research process. It covers roles for patient partners, the research lifecycle, and logistical aspects of including patient partners, such as budgeting and training. Module 2, “Fundamentals of Health Research in Canada”, comprises one three-hour virtual session and 45–60 min of pre-work. It aims to provide participants with a broad understanding of health research: What it is and why it is conducted, who carries it out, how non-researchers can get involved, and some basics about ethics and funding. Completing this module helps participants begin to specify their interests and situate their potential roles in health research. Module three, “Building Partnerships and Consolidating Teams”, also has one three-hour virtual session and includes 45–60 min of pre-work. It helps participants understand the different stages of team development, how teams can be supported through that development, and the meaning of partnerships and group decision-making processes.

Participants

For this evaluation, the online course was delivered seven times, for participants joining from seven Canadian provinces (AB, BC, MB, NB, NF, NS, and YK). The survey response data informing the evaluation presented herein were collected between 2021 and 2023. Course learner groups were generally comprised of one to two research teams seeking to develop their knowledge of and skills in patient-oriented research. Those teams included researchers, patient partners, and others such as clinicians and health policy makers.

Data collection

After the completion of each course module, learners a were sent a post-training evaluation survey via email or QR code. These surveys were based on the evaluation survey for the original Foundations course and adapted by the research team to reflect the virtual format of the course. The email contained a link to the Qualtrics survey platform that housed the survey materials. The survey asked participants to rate each of the three modules on a variety of course elements, as well as to provide written responses to open-ended questions on their experience with and recommendations for the course.

The ratings portion of the feedback survey asked participants to respond to statements about various course elements using a five-point Likert scale of agreement (strongly disagree to strongly agree) and a six-point scale of assessment (very poor to excellent). Completed surveys thereby yielded ratings of both satisfaction with the course and understanding of course learning objectives. Rated elements included pre-work (e.g., “The pre-work exercises were valuable and relevant”); live online sessions (e.g., “The pace of virtual meetings was easy to follow”); co-learning (e.g., “I was able to engage with the facilitators”); and learning objectives, on which participants were asked to rate their knowledge (“e.g., “Defining patient-oriented research and describing how it is different from more traditional health research”). In addition to identifying their role and primary reason for taking the course, the open-ended portion of the surveys sought participants’ views on course take-aways, remaining learning needs, patient-oriented research engagement goals, and recommendations for improvement. Additional commentary on issues not covered by these prompts was also invited.

An evaluation survey was sent to the facilitators after they delivered all the modules for their cohort. The course facilitators were asked to complete a feedback survey that involved rating course elements such as the pace of virtual meetings and opportunities for engagement. Facilitators responded to six statements about their experience (e.g., “During the course, I felt I was able to engage with the participants”) using a five-point Likert scale of agreement (strongly disagree to strongly agree). Facilitator surveys also included open-ended questions on their experience facilitating the course. Questions sought to determine facilitators’ views on how the learner participants managed the material, which areas were most challenging, and what might be improved. As above, facilitators were also invited to share additional thoughts on course matters not covered by the survey prompts.

Data analysis

Data were reviewed, organized, and analyzed using SPSS and NVivo 12 software. Quantitative data were analyzed using descriptive statistics and answers to open-ended questions were examined using qualitative content analysis. For the qualitative data, two members of the research team collaborated to create initial codes and formulate a coding guide. Each research team member independently analyzed the complete dataset, and subsequently the coded materials were combined for comparison. Any disparities in coding were addressed and resolved through consensus discussions. Categories were developed using deductive content analysis and were based on the research questions [23].

Results

Participants

A total of 22 learner participants completed the feedback survey for Module 1. The participant learner group was composed of 8 researchers, 10 patients, and 4 categorized as ‘other role’ (e.g., health professional, ‘other’). A total of 32 learner participants completed the feedback survey for Module 2. The participant learner group was composed of 18 researchers, 13 patients, and 1 categorized as ‘other role’ (e.g., health professional, ‘other’). A total of 24 learner participants completed the feedback survey for Module 1. The participant learner group was composed of 13 researchers, 8 patients, and 3 categorized as ‘other role’ (e.g., health professional, ‘other’). With the total number of course learner participants at 189, the average survey response rate across all modules was 41%. The survey response rate for facilitators was 92%, with 11 of 12 facilitators returning completed feedback surveys. The course was geared towards building capacity in research teams to practice patient-oriented research, with an average of 13 learner participants per class group. In light of these small class sizes and some evidence that survey response rates tend to be lower than 50% [24], the number of completed surveys received seems fairly unremarkable, albeit suboptimal. However, the matter of response rate acceptability thresholds remains an open question, and the commonness of low response rates does not, in and of itself, resolve the challenge of obtaining responses from a sufficiently representative sample.

Learner satisfaction

Survey results indicate a high degree of satisfaction with all measured aspects of the course across all three modules. Learner satisfaction scores organized by module can be found in Table 1. As the data in this table illustrate, participants were overall very satisfied with each course module, with responses demonstrating very similar overall satisfaction with each rated element of the three modules. The only statement about the course that did not receive ratings of agreement or strong agreement by at least 80% of survey respondents concerned the length of live sessions, and even in this case, there was still a strong majority expressing satisfaction.

Table 1 Percentage of participants reporting they agree or strongly agree with statements about the course
Full size table

Learning objectives

Module 1 included 12 items and focused on introductory concepts related to patient-oriented research. The majority of learners reported good or excellent knowledge of the module’s learning objectives. The majority of learners also reported good or excellent knowledge of Module 2’s 10 learning objectives, which focused on health research across Canada. Module 2 objectives were the overall highest rated of the three modules. Module 3 aimed to instill both theoretical and practical knowledge of developing and working within diverse research teams and included six learning objectives. While these received the lowest knowledge ratings among the modules, a small majority of learners still rated their knowledge as good or excellent for most items. Learning objectives ratings can be found in Tables 2, 3, and 4.

Table 2 Percentage of participants reporting good or excellent knowledge of course learning objectives for Module 1
Full size table
Table 3 Percentage of participants reporting good or excellent knowledge of course learning objectives for Module 2
Full size table
Table 4 Percentage of participants reporting good or excellent knowledge of course learning objectives for Module 3
Full size table

Learner group experiences and recommendations

Four main categories were identified through content analysis of the open-ended survey responses provided by course participants: (1) course organization recommendations; (2) course content recommendations; (3) the importance of co-learning format; and (4) the importance of accessibility.

Course organization recommendations

Participants offered recommendations on how to modify the course organization, including reducing the amount of pre-work and adjusting the length of the modules (e.g., Module 1 could be condensed, whereas Modules 2 and 3 could be extended). As one learner surmised, reducing or eliminating pre-work might facilitate participation for a broader range of participants. For example, one learner stated:

“I see this type of pre-work making sense for students or researchers whose job it is to learn about research, but I think it can potentially lead to some people in the group being more informed and taking up more space in the discussions…if this [course] will be done with a cohort with a lot of community members, or front-line health workers, it might be more helpful to try to cover the material ‘in-class’.” (Module 1 feedback)

Others calling for reduced pre-work felt that it overlapped too heavily with the class material, particularly in Module 3. Some participants expressed interest in receiving a breakdown of the course schedule for the day, to have a greater sense of how the session would be structured. Finally, while the co-learning elements of the course were assessed as strong, some participants recommended increased interactive learning in Module 2.

Course content recommendations

Recommendations included more frequent use of concrete, ‘real-world’ examples and activities, working through hands-on problems such as how to complete a project proposal in a team that includes patient partners, and identifying different ways of engaging patients in research. Several participants remarked on the need for problem-solving strategies and conflict-resolution skills, particularly in Module 3, which focuses on team building. For instance, one learner suggested that: “The breakout sessions could have been more reflective of how to improve our team function or problem solving within our team with new strategies to try”, while another remarked that: “Participants would benefit from opportunities to explore real examples of research team challenges and how to facilitate effective team functioning” (Module 3 feedback).

Importance of co-learning format

Participants stressed the importance of including patient perspectives, connecting with others, and reflecting on current research practices. Co-learning involves peer learning in an environment that brings together a multiplicity of individual perspectives and backgrounds. In the feedback for Module 2, which focuses on learning about health research across Canada, participants commented that they appreciated the diverse perspectives and inclusive atmosphere in the virtual classroom. One researcher found that the most memorable part of the session was the “experiences shared by patient partners and their perspective on the research process,” while a patient partner remarked on “the warmth and inclusion aspect where everyone was treated with respect and as an important member of the group” (Module 2 feedback). Participants expressed appreciation for the co-learning format of the course and indicated that this format was key to the course’s success.

Importance of accessibility

Participants commented on areas of reduced accessibility and provided recommendations on how to improve this. For instance, participants expressed interest in more frequent breaks, explicit scheduling and course task reminders, and more use of plain language in the course materials. One learner explained that:

“The use of acronyms, even if explained in the pre-reading, is too much to ask participants to remember. Some of the words used may not be familiar to everyone and could be replaced with simpler words or an explanation.” (Module 1 feedback)

While the virtual course format was generally very well received, with no connectivity issues hindering participation reported, some participants experienced minor technological challenges and others recommended breaking up screen time. As one researcher observed:

“Doing something like this over Zoom can introduce unique challenges that may not occur during in person workshops (e.g., keeping everyone equally engaged), but despite these challenges it was well facilitated and generated an interesting discussion.” (Module 2 feedback)

The course was advertised and delivered entirely in English, so all participating learners and facilitators were fluent in the language of instruction. While this linguistic uniformity streamlined the course adaptation and delivery, increasing broader accessibility will involve expanding the language of instruction to accommodate language diversity.

Facilitator group satisfaction

The majority of course facilitators reported that they agreed or strongly agreed with each of the survey’s statements about the course, as follows: (1) “the workshop materials were well-prepared” (90%); (2) “the facilitation guide provided the information I needed to deliver the session” (90%); (3) “the pace of the virtual meetings was appropriate” (90%); (4) “during the course, I felt I was able to engage with the participants” (100%); (5) “the amount of time spent in the virtual meetings was appropriate” (90%), and (6) “the meetings seemed to flow well” (100%).

Facilitator group experiences and recommendations

Facilitators’ responses to the open-ended survey questions were analyzed and coded into two categories: course organization recommendations and co-learning format. Facilitators’ feedback demonstrated agreement with participant learners on these course elements; for instance, they recommended that the pre-work could be decreased and added to the session material. Facilitators also highlighted the co-learning format as the keystone of the course, due to the importance of hearing patient perspectives, the benefits of a diverse group and shared learning, and the meaningful discussions a co-learning format enables. Some echoed learner participants’ calls for greater focus on instilling practical skills, as one facilitator observed of Module 3: “Our participants were really interested to learn about how to bring together a varied team and get them to a place of high functioning.”

Discussion

This article presents the delivery and evaluation of an online-based patient-oriented research capacity building training program offered to research teams that intend to practice patient oriented research. Research team members enrolled in the course included patients, caregivers, researchers, trainees, clinicians, decision-makers, and other interested individuals engaged in health research. The feedback surveys completed by learners indicated a high degree of satisfaction with the course overall and generally a strong understanding of the learning objectives. Facilitators reported similarly high satisfaction with the course. Both learners and facilitators provided recommendations concerning course organization and content as well. For instance, some respondents identified a need to adjust the ratio of pre-work to online sessions. Learners recommended the inclusion of more hands-on activities to work through planning tasks and identifying different ways of engaging patients in research. They also stressed the importance of making the pre-work and online sessions accessible for all participants. Both learners and facilitators particularly appreciated the course’s co-learning format, which facilitated a great deal of meaningful discussion.

A wealth of literature supports the effectiveness of online learning [21, 22, 25, 26], and in the wake of the COVID-19 pandemic, the virtual delivery of patient-oriented research training courses is becoming more common [e.g., 9, 11, 12, 14, 27]. Ongoing research into optimizing online learning can help guide current and future patient-oriented research training development. For instance, one recent review [20] found that the feasibility and efficacy of online learning are enhanced by topic relevance and acceptability, which can promote engagement and discourage attrition. The current project’s survey respondents demonstrated high engagement with the course materials, with several participants highlighting their appreciation of the course’s interactive elements. Online learning is also facilitated by the availability of user-friendly technology and software that enables high-quality interaction [20]. For learners, the effectiveness of online learning is augmented by a balanced approach that combines independent and interactive activities [28]. Group work in online breakout rooms can be particularly beneficial, as reported by survey respondents and in supporting literature, by fostering collaborative learning, peer-to-peer support, and student empowerment [28]. Online learning offers multifaceted advantages that underscore its suitability as a platform for training diverse audiences in the fundamentals of patient-oriented research.

The recommendations for improvement provided in the survey responses resonate with feedback about other recently developed virtual patient-oriented research training opportunities. For example, Blair et al.’s course, Getting Involved in Research, is also co-created and co-delivered entirely online, with feedback highlighting its co-learning approach as a major advantage [14]. Their course feedback also suggested that participants benefited from interacting over a long period of time; this helped participants to feel comfortable and express themselves openly, as well as to concretize the contributions of their lived experience to research. Similarly, the current surveys’ recommendation for more hands-on experiences has recently been reported elsewhere, and aligns with the objectives of other recently developed courses [29]. For instance, MacArthur et al.’s Patient-Oriented Research Curriculum in Child Health (PORCCH) offers practical skill building for researchers who wish to involve families in their research, but find it difficult to meaningfully engage them and their children [30]. In a similar vein, Blair et al. discuss the value of integrating practical research tasks into their course and having follow-up real-world training opportunities [14]. The researchers note that their aim is not to “professionalise” the skills of those with lived experience, as this may lead to imposing inflexible eligibility criteria and training requirements for research involvement and engagement; rather, they aim to make space for and apply participants’ insights [14]. Many of the current study’s participants shared an appreciation of the fresh perspectives to which they were exposed in the course. That feedback resonates with evaluations of the in-person iterations of the Foundations course, wherein it was determined that a co-learning environment that included representatives from different backgrounds was preferable to a more targeted audience (e.g., only patients) [4]. Another course that focuses on much needed practical patient-oriented research skills is described as a “hands-on training programme” that helps patient partners understand and participate in the grant application process, building confidence and competence in their role in this process [11].

As discussed above, virtual course delivery is well-received and has advantages over traditional learning environments that may be particularly relevant to patient-oriented research capacity building. There is a wide array of factors contributing to the success of a patient-oriented research course, including effective use of ‘real-world’ examples to illustrate key concepts and practices, inclusion of hands-on skill building activities, a co-learning format that encourages high-quality interaction, and accessibility of all course elements to a sufficiently broad set of individuals. The influence of these factors is apparent in the survey response results outlined above. In the following and final sections of the paper, we offer critical considerations of this course evaluation and discuss promising avenues for improving the course, as well as patient-oriented research education and training development more broadly.

Strengths and limitations

The survey design was successful in eliciting participant feedback on a wide range of course elements. The feedback surveys allowed for a mix of response types and encouraged participants to share detailed comments and suggestions on the course matters of their interest. Obtaining feedback from course facilitators provided distinctive insight into how the course was received and a birds-eye view on participant dynamics. The team also faced some challenges, which can be addressed in future research and patient-oriented research training development. For instance, while course participants reported strong understanding of the learning objectives, in the absence of a pre-test survey of participants’ knowledge, we cannot be certain that this is a result of the course. It would, therefore, be desirable to include a pre-test knowledge survey, in order to evaluate the impact of course participation on patient-oriented research skills more effectively. In a similar vein, the current project does not assess the longevity of benefits reaped from course participation. Comprehensive evaluation of training is a vital element of patient-oriented research capacity building strategy and will require tools capable of measuring tangible impacts in research practice. To that end, enhancing future iterations of this course may involve modifying feedback surveys and refining data collection methods such that the course’s practical efficacy can be accurately measured at multiple time intervals (e.g., 3 months, 6 months, and 1 year post-course).

In addition, despite the team’s efforts to attract diverse research teams, very few health system decision-makers and healthcare professionals participated in the course. This warrants consideration of what might have hindered their enrolment, so that recruitment methods may be modified accordingly. Developing new ways of training and diversifying research teams will broaden and refine patient-oriented research capacity. In the same vein, so too will diversifying the course’s language of instruction; offering the course in French, providing live translation in a variety of languages, and providing closed captioning in live sessions would ensure a more fully accessible training opportunity.

Finally, as indicated in the Results section, the response rate for the course feedback survey was lower than anticipated, at 41%. The response rate was below an optimal range, which indicates a need to critically examine each element of the feedback survey’s content, format, and methods of dissemination and collection. That examination should yield understanding of likely contributors to this low response rate, which can then be avoided in future course offerings. For one, future evaluation of this course will be strategized with greater attention to incentivizing survey completion for all participants.

Future directions

Combined with consideration of the relevant literature and similar courses referred to above, this course evaluation provides insight for future directions for patient-oriented research capacity building. Some promising avenues for enhancing patient-oriented research training development have been identified: enhancing practical/applied elements, co-learning and participant diversification, evaluation, and integration.

First, future training development should emphasize the application of learned research skills, in part by offering corresponding training opportunities that are embedded in practice [14]. Indeed, evidence suggests that the majority of learning often occurs in ‘real-world’ settings, and that the Strategy for Patient-Oriented Research’s curriculum will be complemented by a diversity of training resources and opportunities [4]. There is good reason to offer tandem or follow-up on-the-job training for interested participants.

Second, a greater emphasis on participant diversity is needed to optimize capacity building in patient-oriented research. Barriers to participation among groups with lower representation in patient-oriented research training, such as clinicians and health system decision-makers, warrant closer attention. The survey responses indicated lower participation and satisfaction from these groups, yet their involvement is key to robust capacity in patient-oriented research, as well as to its effectiveness in supporting the shift toward patient-centred care. The exceptionally demanding schedules of clinicians and health system leaders present an ongoing challenge for their participation in time-consuming research and training. Issuing sufficient time for these professionals to play an active role on research teams and to complete patient-oriented research training may be facilitated via formalized incentives such as continuing education credits. Further research into participation barriers and possible strategies for surmounting them will set the stage for meaningful participation incentives that will support broader representation in patient-oriented research.

Third, there is a growing need for standardized methods of evaluating patient-oriented research curricula. The current course evaluation and others like it found it difficult to assess learning outcomes and usability of patient-oriented research training [12, 14]. More broadly and despite the rapid proliferation of patient-oriented research capacity building initiatives, measuring the impact of those initiatives will require developing new tools, which will be key to advancing patient-oriented research skills [31].

Conclusions

This article outlined the design, methods, and findings of a survey-based course evaluation examining participants’ reception of a co-designed and co-delivered online course in patient-oriented research that was adapted from the Canadian Institutes of Health Research Strategy for Patient-Oriented Research’s in-person Foundations course. As indicated above, participants included patients, researchers, clinicians, health system decision-makers, and others involved in health research. Course feedback surveys indicated that the course was well received by all participants, demonstrating a highly successful adaptation from in-person to virtual delivery. Some recommendations for improvement were also provided by participants, and will be implemented in future versions of the course. Patient-oriented research reflects a broader shift in the health research landscape across Canada and beyond; patient partners and other non-researcher team members are now recognized as part of a collective that works together at every stage and in each aspect of health research [31]. The positive assessment of this course is thus a welcome result not only for study participants, but for anyone involved in or impacted by health research in general; fostering capacity in patient-oriented research via accessible training opportunities is critical to realizing the many benefits of engaging patients in health research. Supporting the proliferation of patient-oriented research teams requires ongoing opportunities for training and development, which in turn requires building connections across researchers, patients and families, caregivers, health care professionals, and health system decision-makers. Initiatives such as the Canadian Primary Care Research Network and the aforementioned Strategy for Patient-Oriented Research have been building communities of practice that enable research teams to come together, develop their skills, and apply their growing expertise to carry out impactful patient-oriented research.