Providing graduate students and postdoctoral scholars with tools, resources, and mentoring to support their success early in their careers (Davis, 2005; Fuhrmann, Halme, O'Sullivan, & Lindstaedt, 2011; Helm, Campa III, & Moretto, 2012) is essential, particularly since students and postdocs often change their professional goals as they acquire more information and experiences (Fuhrmann, 2016). One aspect of effective support is offering professional development activities, which are often components of individual development plans. Establishing resources for professional development has become a national priority; and several entities support these efforts including but not limited to Broadening Experiences in Scientific Training programs (Meyers et al., 2016), the National Research Mentoring Network (Lee, McGee, Pfund, & Branchaw, 2015; Williams, Thakore, & McGee, 2015), iBiology (Goodwin, 2014), and the Council of Graduate Schools (Denecke, Feaster, & Stone, 2017).

The institutions of the authors of this article are members of The Center for the Integration of Research, Teaching and Learning (CIRTL) Network (Austin et al., 2009; Hill & Austin, 2016). The mission of CIRTL is to improve the quality of Science, Technology, Engineering and Math (STEM) undergraduate education through the development of a national faculty committed to evidence-based teaching practices for diverse learners. To accomplish this goal CIRTL focuses its professional development efforts on future faculty, namely STEM graduate students and postdoctoral scholars. CIRTL began in 2003 and has built a national learning community by offering synchronous online professional development programming (Austin et al., 2009; McDaniels, Pfund, & Barnicle, 2016; Pfund et al., 2012). These professional development offerings range in terms of participant time commitment and engagement from informational webinars to semester-long courses and annually reach approximately 1100 participants from 38 CIRTL Network institutions and 92 other non-member institutions nationally.

Though CIRTL had been offering many other types of professional development programming, until 2016 CIRTL had not offered an online synchronous workshop format as a means of professional development for graduate students and postdocs. The authors of this article, as members of the Network, designed a workshop structure and piloted four distinct workshop offerings to CIRTL participants using the new format (Table 1). Workshops were 2 to 4 hours in length, over one or two sessions. Our goal was to develop and assess a structure for professional development workshops to maximize our participants’ skill development and their commitment to follow-up actions or developing new behaviors. During these four workshops instructors and participants engaged in discussion, brainstormed, and reacted to the content. They also created and shared their work with their peers, providing feedback and reflections through a synchronous platform (Blackboard Collaborate). In the process we created an atmosphere of collaboration, trust, and shared ownership of the learning process (Gunawardena & Zittle, 1997; Wei & Chen, 2012).

Table 1 Workshops developed using active learning framework
Full size table

Although our design focused on helping participants create specific materials related to our learning outcomes (e.g., an individual development plan), we found through the study of the four pilot workshops that what was most valuable to participants were the opportunities for community feedback and personal reflection during the online sessions. In this article we explain our framework for online workshop design, describe our implementation and evaluation, and also suggest how institutions can adapt this structure to appeal to their own graduate students and postdocs to meet their professional development needs.

Workshop Design and Approach

Through fall 2015 and spring 2016 we piloted synchronous, online, CIRTL-sponsored workshops on four different topics reflecting the elements of our new design. This design is based upon an inform-create-evaluate (ICE) structure (Hokanson, Campa III, & Goldberg, 2017):

  • Inform: Workshop content is aligned with learning objectives. Background content is explored by participants through pre-session exercises and through early in-session inquiry that is guided by facilitators.

  • Create: Participants create materials associated with the session topic (e.g., a plan, a writing assignment, an action-based reflection), allowing them to gain knowledge and demonstrate their developing professional skills.

  • Evaluate: Peer feedback about participants’ work targets high order learning objectives. Peer assessment also provides a way of measuring participants’ learning by evaluating learner-created materials. Participants are provided with rubrics to inform how they deliver feedback and are placed in small groups to limit the amount of effort associated with reviewing the work produced by their peers. This approach allows evaluation of learner-created materials to scale to larger workshops.

Though the total participation in CIRTL programming is much larger as described above, our workshop pilots reached 240 geographically distributed participants from 21 universities. Participants for the workshops were recruited from STEM Ph.D. students and postdocs at the CIRTL Network member institutions through email and newsletter communications directly from CIRTL and from professional development offices within those institutions. The workshops were also advertised on the CIRTL website, which attracts non-member participants interested in career and professional development.

We intentionally selected workshop topics that focused on some of the skills needed for faculty career readiness, specifically the skills of career planning, resilience, building teaching and research learning communities, and writing a statement of teaching philosophy. In our design we also shifted focus away from content absorption towards creation, interaction, feedback, and reflection. Where possible, we sought to make connections among the learning outcomes by providing strategies for participants to apply their new skills after completion of the workshop(s).

Participants registered for workshops through the CIRTL Network website and were given access to a workshop-specific website, which contained all pre-session resources and assignments. For this study we took advantage of the learning management system, Moodle, which provides instructors the capability of communicating with participants and the ability to collect and distribute resources, post reflection prompts in discussion forums, and collect learner-created materials and assignments in advance.

The workshops took place on the Blackboard Collaborate platform. On this platform participants are able to (a) hear, see, and interact with their facilitators and one another; (b) work together in virtual breakout rooms by writing on shared Google Docs or whiteboards; (c) engage in written (e.g., “chat”) and verbal discussions; and (d) provide feedback to one another on written documents. Facilitators were from CIRTL Network member institutions and included Ph.D.-level academic administrators with expertise in professional development and teaching and learning, as well as STEM tenure-line faculty members. In addition, multiple facilitators were involved in leading each of the four workshops. This ensured that a range of disciplines and expertise were represented and that multiple modes for simultaneous interaction (e.g., verbal, chat/text) were monitored and incorporated into the discussions. All workshops contained design elements of peer-to-peer interactions in small group breakout sessions with between 4 and 15 participants in each group.

Based on the literature and recommendations from leaders in graduate education (e.g., Council of Graduate Schools) and postdoctoral training (e.g., National Postdoctoral Association) we selected topics that developed core skills necessary for a future faculty career. We also considered what topics would be broadly appealing to graduate student and postdoctoral scholar audiences based on our prior experience developing workshops within our local institutions. Developing Work-life Resilience (WLR) was a single session, 90 minutes workshop offered once during this study. Its content focused on developing strategies to help participants deal with and rebound from stressful situations, which are particularly important for early career faculty members so as to help them successfully navigate the multiple stresses in the tenure track process. Creating and Owning Your Individual Development Plan (IDP) was a 90 minutes, single session workshop focused on developing a career plan that incorporated self-assessment and reflection on skills, interests, and values. It was offered twice during this study. Writing a Teaching Philosophy Statement (TPS) was offered as a two session workshop (180 minutes total) that focused on developing and reviewing draft teaching statements to be used in applications for faculty positions, and it was offered twice in this study. Creating Effective Learning Communities (LC) was offered once during the study as a two session workshop (180 minutes total) and focused on developing the leadership and teaching skills related to leading research- or teaching-based learning communities.

We provide a representative workshop design in the form of a planning grid from our WLR (Work, Life, Resilience) workshop (Table 2). As shown in the grid, the WRL workshop follows the inform-create-evaluate (ICE) structure. For example, in the work particiapants completed prior to the WLR workshop, they reflected on their prior knowledge about resilience and framed larger issues or personal challenges through discussion forums and reading materials provided by the facilitators (i.e., inform from the ICE model). Participants also watched pre-session videos that featured a panel of workshop facilitators discussing prompts that would be used in-session. This pre-session work initiated participants’ engagement in the workshop and set expectations for their full participation during the synchronous session, including use of their video and microphone as well as typing responses in the chat window or on provided whiteboard slides.

Table 2 Sample workshop development template for Developing Work-Life Resilience
Full size table

During our synchronous sessions participants practiced and applied new skills through interactive activities, and they provided and received peer feedback to assess their progress. For example, in the IDP (Individual Development Plan) workshop, participants used a grid to explicitly build connections among their skills assessment, career aspirations, and short- and long-term goals; and they identified resources they could use (i.e., create from the ICE model). In the other workshops additional activities included case study analyses and discussions, writing assignments, brainstorming, analyzing observations of practices related to the workshop, role-playing, and/or viewing instructional videos followed by reflection prompts and discussion. The four workshops also included developing action plans as a creation step (create, ICE model; WLR), teaching statements (Creating an Effective Teaching Statement [TPS]), and designing an activity to build or support a learning community (Creating Effective Learning Communities [LC]).

During the workshops we completed the evaluate portion of our inform-create-evaluate framework using self- and group reflection exercises and peer assessment. Following the workshop, we then conducted a program evaluation of the framework as a whole, as described below. The evaluate part of the framework was designed to motivate participants to continue their learning or skill development post-workshop and to create connections to their desired career pathways. For example, during the learning community (LC) workshop participants used a facilitator-developed rubric to assess a learning community at their local institution (evaluate in the ICE model). Activities or content associated with these workshops are accessible through requests made to the first author of this article.

The Study

Purpose

We conducted an evaluation of the workshops with three goals in mind. First, we aimed to obtain feedback on the workshop framework to increase the potential for learning and change. Second, we wanted to measure the impact of the workshops on participants’ knowledge, perceptions, attitudes, and behaviors. Working within our intended outcomes, we were particularly interested in the impact on the behaviors of continued work on the learner-created materials and discussions about those materials with mentors. Finally, we sought to understand the experiences and impact of the workshops beyond the framework of our own intended outcomes. We gave deliberate attention to participants’ desired outcomes (which Ricardo Wilson-Grau refers to as “harvesting” participants’ outcomes, (Wilson-Grau, 2019)) and the degree to which their own outcomes were achieved. This required considering students’ broader experiences including their professional development and mentorship opportunities and experiences.

Boston University’s Internal Review Board reviewed the study of the workshop participants (protocol number 4046X; February 18, 2016). This review determined the study was exempt from IRB review.

Sample

We conducted two surveys in order to assess our first two goals of measuring impact and to obtain feedback on the model. We distributed surveys to the full population of 240 workshop participants, who were graduate students (57%, n = 137) and postdocs (33%, n = 78), and staff (10%, n = 25). We surveyed them immediately following the workshop and then again between 6 and 18 months subsequent to their participation. The average response rate for surveys administered immediately following a workshop was 56%. The follow-up survey garnered a 31% response rate, which is typical of similar populations (Sauermann & Roach, 2012).

To address our third goal of understanding students’ experiences more holistically, including their own intended outcomes for the workshop experience, we conducted individual interviews after the second survey. Using a purposive sampling strategy, we targeted both workshop participants whose survey data suggested that they had not completed the learner-created materials and/or had not experienced changes in perspective, in addition to those who had. We recruited from a list of survey respondents who consented to be contacted for interviews. The eight interviewees we selected represented a mix of postdocs and graduate students, participation across all four workshops, and a fairly even gender mix. Each interviewee had taken between one to three workshops. Three interviewees had taken Creating and Owning Your Individual Development Plan (IDP); four had participated in Writing an Effective Teaching Statement (TPS); two had taken Creating Effective Learning Communities (LC); and three had taken Developing Work/Life Resilience (WLR).

Our sample size allowed us to achieve our objectives and data saturation. Qualitative samples tend to be small because the phenomenon of data saturation is often acheived early and larger samples yield diminishing returns, particularly in cases like ours where there is reasonable homogeneity within the group and the method employed is one-on-one interviews. We were not seeking data about prevalence, and the sample needed to remain somewhat small in order to do justice to our detail-rich data (Ritchie, Lewis, Nicholls, & Ormston, 2014).

Data Sources

Surveys

We administered the survey immediately following the workshop and the follow-up survey using Qualtrics, an online survey platform. Survey instruments included closed- and open-ended questions and took between 5 and 15 minutes to complete. The follow-up survey included a question about the respondent’s status as a graduate student or postdoc to gauge the potential influence of academic status on learning and behavioral outcomes.

Post-workshop survey questions were tailored to each workshop based upon the stated learning objectives. For example, respondents were asked to rate their familiarity with specific concepts both before and after participating in the workshop. We assessed learning outcomes primarily through self-report although we also included more direct assessment questions (e.g., “Rate the importance of the following characteristics to creating successful learning communities”). The surveys evaluated behaviorial shifts through questions about participants’ actions related to the topic and the materials they created during the workshop. These materials varied according to the workshop (e.g., “Have you ever completed an IDP and discussed it with your mentor?”). Program evaluation questions were consistent for all four workshops and were directed toward the workshop design (e.g., length), quality of facilitation, value of peer interactions, and program engagement. The surveys did not include explicit questions about opportunities for reflection because we had not developed specific hypotheses about their value prior to conducting interviews.

Interviews

We developed a flexible interview protocol that allowed us to explore what participants were hoping to gain from participation and what they found valuable about the workshops. To evaluate the degree of and reasons for change in behaviors, perceptions, and attitudes, we needed to examine the range of participants’ experiences. Therefore, the interview protocol addressed the motivation to register, expectations of the workshops, and students’ perceptions of the impact of the workshop(s). While survey respondents could tell us whether they experienced behavioral and attitudinal changes as defined by workshop designers, interviewees could help us understand why they may or may not have experienced those changes, other ways they may have been changed by the workshops, and how those changes aligned with their own expectations and needs. As part of our effort to understand the interviewees’ complete experience, we also asked how faculty mentors and institutions influenced their engagement in professional development. Our questions gave interviewees the opportunity to explore the themes of reflection and community more intentionally than in the survey.

A co-author of this paper who was independent of the design, implementation, and facilitation of the workshops conducted the one-on-one telephone interviews. Using a neutral interviewer allowed us to ask potentially sensitive questions about participants’ workshop experiences, post-workshop behaviors, and interactions with their mentors. Interviews ranged from 25 to 45 min in length and followed a flexible protocol that was guided by the broad topics mentioned above and tailored to individual workshops taken and interviewees’ follow-up survey responses. We offered an incentive of a $10 Amazon e-gift card to participants to help mitigate sampling bias.

Data Analysis

We analyzed post-workshop data from each of the four workshops independently. Given the variability in response rates and the generally low sample sizes (ranging from 8 to 84 responding participants from each offering of the workshops), we analyzed the quantitative data conservatively using descriptive statistics to summarize data from Likert scale items. Although we compared data across different workshops, we avoided using inferential statistical tests. In addition, we applied thematic analyses to evaluate the qualitative data that were collected with open-ended questions (Braun & Clarke, 2006).

The co-author who conducted the interviews performed an interpretative phenomenological analysis (Tuffour, 2017) of those data in order to identify recurring themes. A comparative analysis (Patton, 2002) revealed common experiences across workshop types and experiences unique to specific workshops.

Findings

Though the participants we surveyed and interviewed had participated in four different workshops and had a wide range of backgrounds, prior experiences, and perspectives, consistent themes emerged from our data, particularly during the interviews. Overall, survey findings indicated participants were satisfied with the workshops; across all four workshops the majority of the respondents (79%; n = 86/109) would recommend the workshop to their peers. Nearly two-thirds (61%; n = 56/92) felt that the workshop length was appropriate; roughly a third (32%; n = 29/92) thought they could be longer. Workshop facilitators also received high ratings for their effectiveness (70%; n = 61/91 indicated they were extremely or very effective). See Table 3 for findings on program components by workshop type.

Table 3 Post-survey data from graduate students and postdocs workshop participants
Full size table

Our findings from the surveys suggested that participants greatly valued the workshops, but not necessarily for the opportunity to produce materials as we had initially hypothesized. For example, while over half (58%; n = 10/17) of those participating in the Writing an Effective Teaching Philosophy Statement workshop had revised their statement draft at the end of the workshop, only a third (35%; n = 6/17) had discussed it with their mentor. At the time we implemented the follow-up survey, only 30% (n = 10/33) had made use of their philosophy statement for purposes of career advancement. We inferred that our workshops were accomplishing participants’ self-defined goals for participation, which were not always associated with the materials that the participant had developed in-session (Table 1).

Although the surveys did not explicitly ask about community building, they did include questions about opportunities for peer interaction. Survey respondents considered facilitated and non-facilitated peer discussion groups to be very helpful (Table 3), and discussion groups featured prominently in participants’ responses to other open-ended survey questions (Table 4). When survey respondents were asked an open-ended question about when they felt most engaged or what was most helpful, 68% (n = 34/50) referenced peer interactions; many described the benefits of these interactions in detail.

Table 4 Responses to peer discussions from open-ended survey questions, with representative comments (RC)
Full size table

As a result, we suspected that community building may have been more valuable to students than the learner-created materials they produced. We sought to understand more fully how participants defined their own objectives, the impacts of the workshops that they experienced, and the perceived value or role of the learner-created materials. We used the interviews to address these evaluation questions.

Interview findings did, in fact, suggest that participants viewed the workshop content and learner-created materials differently than we intended. For example, interviewees often signed up for workshops that explored topics they had already thought about deeply and about which they will continue to think about on their own and with peers. In other words, the workshops represent one moment in a larger developmental process; and the impact is difficult to separate from this cumulative, constructivist learning process. Similarly, the use of learner-created materials were perceived as part of an ongoing process, rather than a one-time outcome to be used or implemented. For example, when an IDP workshop participant registers for the workshop, she may have already established some goals, may further develop these into a plan during the workshop, and continue to revisit and rewrite her goals after the workshop, regardless of whether she formally follows up in the ways intended through the workshop design (e.g., mentor discussion). In reality, she may not speak with her mentor about the specific IDP she created, but the workshop may prompt her to find alternative ways to discuss her professional goals with her mentor. This reality and complexity of experience made it difficult for participants to attribute attitudinal changes to participation in a workshop. Additionally, this conclusion means a survey question designed to document a narrowly-defined and time-bound behavioral change will not capture the range of potential behaviors nor the value assigned to workshop content and materials produced.

Interviewees reported using workshops as opportunities to reflect on their career trajectories and values. They often took workshops at transitional moments in their lives and careers, and in the absence of supportive mentorship from their research mentor(s). Additionally, participants reported finding it difficult to make time for this needed reflection without the structure afforded by something like a workshop. In unprompted comments, five of the eight interviewees described the significance of attending workshops for setting aside time to reflect on their careers and goals. The three other interviewees, who did not mention this element, stated that they had multiple opportunities for professional development.

Interviewees reported that they needed the structure the workshops provided to pause, reflect, and discuss their careers; and they described taking their professional development into their own hands. They spoke of taking comfort in learning that other participants have faced challenges similar to theirs, both with advisors and more generally in their academic lives. Additionally, interacting with peers from other institutions was particularly valuable for those in more remote geographical locations. Interviewees also expressed a strong interest in continuing these conversations beyond the workshop, perhaps in an online forum, to follow up with new colleagues and expand their professional networks.

Both survey and interview data showed that participants were most engaged during the peer interaction and feedback portions of our workshops. Seven of the eight interviewees similarly described the positive impacts of hearing from and interacting with peers during the workshops. However, workshop design elements impact how valuable small group and peer-led discussions can be for participants. Also, many would have liked to have been in breakout groups with true “peers,” such as postdocs with postdocs, and/or matched up with participants in their disciplines. While some interviewees spoke of the benefits of learning about the varieties of experiences of their peers, they also felt that the differences between postdocs and graduate students in particular made conversations and peer feedback more challenging.

Three interviewees commented that they benefitted from their observations of the workshop instructors modeling active learning and engagement in an online environment. For example, one interviewee said:

I’m amazed at what you can do in that online format if it’s done well. I never would have thought it would be as effective as I find them to be in an online class. I’m super impressed, and it’s totally going to change my mind about what you can do with the online course.

Others found the design of the workshop helpful for identifying strategies to engage students in their classroom: “[By observing, I learned] how to motivate people to engage in discussions.”

Discussion

Our inform-create-evaluate approach is a framework for workshop planning, implementation, and evaluation. Participants chose to participate in our workshops because they needed to complete a task or an assignment (e.g., develop an IDP) and because they wished to take ownership of their own professional development and career trajectory. They selected topics upon which they wanted to reflect, making time to do so amidst day-to-day research and teaching duties. Our workshop activities served as a vehicle for them to benefit from peer and instructor feedback. Post-workshop, participants adjusted the way they approached their mentor or advisor on subjects related to the workshops or how they thought about their career planning.

Based on our findings from this pilot project, we have now adapted our framework to incorporate more opportunities for participants to reflect and get feedback at each learning stage and to connect to the CIRTL community of participants and facilitators. Pre-session, we primarily engaged participants individually through self-paced assignments and pre-surveys. Going forward, we will compliment these with pre- and post-workshop discussion prompts and narrative sharing with peers. We will also share aggregate participant pre-survey data as a way of inviting conversation prior to the workshops. Our data suggests a workshop is one moment in a larger dialogue focused on a particular topic, and pre-survey prompts for discussion may provide insight into what peers have been thinking about or already doing on that topic.

Our workshop sessions were largely centered on the assignment being developed as a driver toward skills application and building competency. We now know that the materials participants produce is only one aspect of their skill growth and career preparation within a broader professional development trajectory. Revisions to our offerings will include connecting their work to broader career goals. We also plan to provide more opportunities post-workshop for participants to continue their progress. For example, we plan to expand our end-of-session exercises to prompt participants to reflect more deeply on how they plan to incorporate the feedback they received from their peers and integrate what they have learned into their day-to-day practice – in essence, defining their action items. We will also invite participants to explore options that will sustain the online community created during the workshop.

Our findings also suggest the importance of creating active engagement in the online synchronous setting. Content learning goals vary based on the topic and audience, yet the consistent value placed by our participants on learning goals around reflection and community indicate the high value of thoughtful discussion, broad participation, rapid brainstorming, and personal reflection. These goals then inform the strategies used during the workshop such as polling, sharing written comments on a whiteboard, use of breakout rooms, or verbal discussion, each of which may support varied degrees of anonymity, depth, pacing of presenting and discussing, reflection, interaction, or vulnerability. Intentional backward design (Wiggins, 2006) from goals to strategies is critical in the online setting where social interaction can be challenging, face-to-face cues are absent, a sense of all being present can be more elusive. We know that there are multiple strategies to facilitate interaction in the online environment.

We supplemented our pre- and post-surveys with interviews as a way to further our understanding of participants’ learning. While not always feasible for programmatic evaluation, we found that this approach to evaluating a pilot program can address gathering nuanced feedback from survey data. We will use redesigned survey instruments for future workhops, which will allow us to evaluate students’ needs and interests more effectively and understand the impact of our work. A common set of updated survey questions to make comparisons among workshops is now used by the CIRTL Network for evaluating cross-network workshops and is available through requests made to the first author of this article.

Conclusion

Participants in our synchronous online workshops indicated that completing a tangible assignment and building a sense of community through structured time for reflection were the principal elements of our ICE workshop design that they valued. One of the key challenges in a synchronous online setting, with a one-time event where participants are likely not to know one another in advance, is to create a sense of community to support participant collaboration. We implemented workshop design elements to foster engagement among participants in the online environment. Our survey and interview-based findings inform our understanding of effective workshop design that includes active engagement, time for reflection, creation of a product, and the prioritization and intentional design of structured time for engagement with peers. Our findings suggest that intentionally structuring time for participants to share ideas, receive peer feedback, and learn by listening to others should all be key parts of the online professional development workshop design for such events.