Towards scaffolding self-regulated writing: implications for developing writing interventions in first-year writing

Abstract

Writing is a crucial, interdisciplinary skill that incoming college students need to successfully complete many of the tasks assigned within their coursework. While teaching self-regulation skills for writing has become more commonplace in writing curricula, and research has investigated how students have been impacted by a writing-about-writing curriculum or how they write in the classroom and perceive their use of self-regulatory strategies, there is not as much research combining these approaches. We argue it is important to investigate the ways that students are impacted by this curriculum and how they perceive their own self-regulated writing behaviors—particularly as evidence by how students write in real-time using multimodal data channels. As such, one goal of this paper is to highlight our approach to investigating college students’ (n = 62) writing processes as they responded to a written self-reflective prompt. Based on students’ written responses, we discovered four student clusters and compared their keystroke-logging behavior using a one-way MANOVA, with post-hoc analyses revealing significant differences between production and revision behavior between some clusters, but no differences in pausing behavior. In addition, another goal of this paper is to derive recommendations for the design of scaffolding based on the results of this empirical study, which imply that scaffolded support should focus on particular phases of self-regulation for different groups of students. Future studies are needed to test the most beneficial ways to scaffold students to ensure they are engaging in effective writing strategies that promote higher levels of metacognitive awareness of one’s writing to ensure students are effective writers throughout their years in college and beyond.

Introduction

Writing is a crucial, interdisciplinary skill that incoming college students need to successfully complete many of the tasks assigned within their coursework (e.g., writing essays, drafting lab reports, submitting research proposals, etc.) and eventual careers. Transitioning to post-secondary level writing assignments (and associated expectations) can be challenging, especially when students lack the awareness of how to regulate their writing processes in a way that may transfer across academic writing tasks (Downs & Wardle, 2007; Graham, 2018; Negretti, 2012). In response to these concerns, college composition programs throughout universities in the United States have adopted curricular approaches that have adapted self-regulated learning (SRL) strategies to foster metacognitive awareness about writing, such as the Writing-about-Writing (WaW) curriculum (Downs & Wardle, 2007). Such an approach has demonstrated success through engaging students in scaffolded writing activities, such as researching writing processes as a phenomenon and reflecting on their own writing processes (Wardle, 2013). In other words, self-regulated writing, like SRL, may empower young writers to take control of their own learning and writing processes.

While self-regulated writing may have demonstratable benefits for writer students, Graham et al. (2018) have nonetheless argued that less is known about “how to teach students to use self-regulation to enhance their writing performance” (p. 146). Further, by the time that students are enrolled in college, their experiences writing in K-12 may have cemented how they value and perceive their own writing skills and processes, which may make it difficult to foster new strategies (Ekholm et al., 2015). We therefore argue that in order to research best practices for teaching writing instruction, particularly scaffolding for self-regulated writing, it is important to better understand the self-regulated writing strategies students already use and—more importantly—their perceptions about their writing processes. Our study is therefore concerned with understanding the relationships between college students’ self-reported writing processes and attitudes about their writing processes in order to inform scaffolding for self-regulated writing. To do so, we employed mixed methods to both collect and analyze multimodal data channels. In this paper we report on both qualitative analysis of student composed self-reflections of their writing processes, as well as quantitative analysis of keystroke log-file data.

Theoretical framework

The current study utilized Graham’s (2018) Writer(s)-Within-Community Model of Writing to examine qualitative and quantitative data produced during a self-reflective writing task. Graham’s model builds upon existing cognitive models of writing (Hayes, 2012; Leijten et al., 2014) that consider how writers simultaneously use multiple cognitive and psychomotor mechanisms during writing, and socio-cognitive theories of self-regulated learning (Winne & Hadwin, 1998, 2008; Zimmerman, 1986) that consider individuals as active participants in their own learning through multiple recursive phases. We therefore chose Graham’s model because it provides a theoretical lens that allowed us to consider the multifaceted, inter-woven social and cognitive mechanisms involved when writers compose and revise texts.

Writer(s)-within-community model

As writing involves a complex interplay between social, technological, and cognitive processes (Hayes, 2012), writing can be characterized as applied metacognition (Hacker et al., 2009). Graham’s model builds upon self-regulated learning (SRL) theory (Zimmerman & Risemberg, 1997), theories of social cognitive learning (Bandura, 1988), and cognition and writing (Hayes, 1996, 2012; Leijten et al., 2014). Graham’s model merges these perspectives to generate a model with which to study the interwoven social and cognitive processes driving writing behaviors within writing communities (i.e., classrooms) and individuals (i.e., first-year writing students; Graham, 2018). According to this model, cognitive processes involved during writing are embedded in social contexts (i.e., socially relevant genres and writing tools for writing) and draw upon individual and social knowledge (i.e., long-term memory as well as community beliefs) in order to complete writing tasks (Graham, 2018). For example, a writer composing an email will draw upon different knowledge about the form and content of the email as it relates to who is going to read the email and what actions it is going to accomplish. Drawing upon the Hayes model (Hayes, 2012; Leijten et al., 2014), Graham also argues that these writing processes would be evident in the way the writer composes text for that email.

As noted above, a central tenet of Graham’s (2018) model is that writing is always embedded within and motivated by real-world social context. Writers are not lone wolves—they have learned to write and purposefully engage in writing tasks as part of community contexts that may involve an array of individuals—including teachers. Therefore, while writing certainly requires an individual’s cognitive processes to generate ideas, transform those ideas into language, and transcribe them onto paper, the activation of those cognitive processes to produce written text requires a multitude of social and community actors to teach various writing skills, processes, and strategies.

Literature review

In addition to theory, this study was also inspired by ongoing scholarly work in self-regulated writing. These studies have investigated strategies for self-regulated writing (Graham et al., 2018; Zimmerman & Risemberg, 1997), how students perceive their self-regulated writing and metacognition (Negretti, 2012; Zumbrunn et al., 2016), the outcome of self-regulated writing instruction in writing classes (MacArthur et al., 2015) and how students engage in self-regulated writing (Bai, 2018).

Negretti’s (2012) semester-long study collected data through journaling in order to identify metacognitive thought and awareness through reflection. Results from the student participants (n = 17) of various ages at a community college found that expressions of metacognitive awareness corresponded to high writing self-efficacy and a connection with the intended audience of the writing (Negretti, 2012). Echoing Zimmerman and Risemberg (1997), this suggests that a writer’s self-efficacy about their writing, perhaps as developed through formal writing instruction and through feedback from peers and experts, may simultaneously foster improved metacognitive awareness and self-regulated writing. Zumbrunn et al. (2016) explores this connection by identifying the role of feedback perceptions and a student’s ability to self-regulate during a writing task. In a study of almost 600 middle and high school students of varied backgrounds across four different schools, Zumbrunn et al. (2016) indicates that those with positive perceptions of feedback identified as having higher aptitudes of self-regulation, and therefore can potentially provide an environment that produces better student writers.

Furthermore, analysis of think aloud protocol data collected during writing from primary school students (N = 32) in English-as-Second-Language (ESL) writing classes found evidence that student writers indeed engage in self-regulated writing strategies during writing (Bai, 2018). Bai’s (2018) study additionally found evidence that suggested more advanced writers engaged in self-regulated writing more so than less advanced writers, similar to Negretti’s (2012) observations. This research suggests that self-regulated writing may have an array of benefits for writers, and that use of self-regulated strategies may be impacted by writing instruction as well as writers’ perceptions about themselves and others.

First-year writing programs utilizing WaW curricula have demonstrated successes increasing students’ metacognitive awareness of their own writing through explicitly teaching students how to effectively self-regulate across academic writing tasks (Downs & Wardle, 2007; Negretti, 2012). In practice, this often involves direct instruction on self-regulated writing strategies such as planning, monitoring, and reflecting while composing and revising texts (Graham, 2018; Winne, 2018; Zimmerman & Risemberg, 1997). Research has investigated writing in terms of how students have been impacted by a WaW curriculum (e.g., Downs & Wardle, 2007) or how they write in the classroom (e.g., MacArthur et al., 2015) and perceive their use of self-regulatory strategies (Negretti, 2012). However, there is not as much research combining these approaches to investigate the ways that students are impacted by this curriculum and how they perceive their own self-regulated writing behaviors—particularly as evidence by how students write in real-time using multimodal data channels.

Scaffolding self-regulated learning and writing

Several studies have examined both the theoretical development of scaffolds designed to prompt effective SRL behaviors and the impact that this form of scaffolding has on learners’ use of self-regulatory strategies and subsequent academic outcomes (Azevedo et al., 2011, 2016, 2022; Johnson, 2019; Su, 2020). Research about the effectiveness of teaching self-regulatory strategies in writing classrooms, i.e., self-regulated writing, has had positive benefits as well. One approach, called Self-Regulated Strategy Development (SRSD), explicitly teaches self-regulated writing strategies for various stages of writing such as planning, drafting, and revising (Graham et al., 2018). While aimed at K-16 students with learning disabilities, SRSD has demonstrated significant improvements to all student writing, including writing quality and knowledge, use of planning and revising, and even writing self-efficacy (Graham et al., 2013).

Research in secondary education has also demonstrated the effectiveness of various forms of writing instruction. For example, Galbraith and Torrance (2004) found that revision processes may differ depending on the writing context and even stage of drafting, and that more structured planning strategies (i.e., outlining) may improve textual quality. Vandermeulen et al., (2020a, 2020b) found that synthesizing multiple sources in a text may depend on genre, prior knowledge, and reading strategies. Vandermeulen et al. (2023) found that use of writing process reports (Vandermeulen et al., 2020a) can offer students individualized feedback about their writing process. Further, Vandermeulen et al. (2023) observed that asking students to reflect on their own writing process report in comparison to that of a writer who produced a higher-rated text led to improved revisions. Collectively, this body of research suggests that scaffolded instruction for planning, reading strategies, and receiving and reflecting on feedback can significantly improve student writing at the secondary school level.

Research in post-secondary school has also shown the benefits of self-regulated writing. MacArthur et al. (2015) studied the effect of explicit self-regulated writing instruction in first-year writing courses at two universities. Results from their quasi-experimental study found that the writing quality of student participants (N = 276) benefitted from learning about self-regulation and writing. Wardle (2013) has additionally demonstrated that writing instruction that directs students writing to researching writing as a field of study (i.e., WaW), has positively affected student achievement of writing outcomes and attitudes about their own writing. Such success is typically achieved through a highly structured, semester-long research project that guides students through various aspects of secondary and primary research, reflecting on writing and research, drafting, providing feedback, and revising writing.

Although limited, these empirical studies demonstrates that self-regulated writing may indeed be fostered in a writing community such as a classroom, that it is closely related to writers’ self-efficacy, and that explicit scaffolding within the writing process itself may benefit writers’ use of self-regulated writing (Graham et al., 2018; Zimmerman & Risemberg, 1997). Nevertheless, the effects of specific instructional strategies to scaffold writing students toward improving their writing and use of self-regulated writing is underdeveloped. As Graham et al. (2018) note, SRSD instruction prioritizes individual student preferences and needs, which requires understanding students’ perceptions about their own writing processes in order to design scaffolding to intervene. As will be discussed in the following sections, one of our study’s broader goals is to use multimodal data channels to develop real-time interventions for student writers. How can we develop individualized scaffolding through understanding student perceptions about their own writing as well as their real-time writing behaviors? Increased attention on the importance of SRL and how learners deploy SRL strategies in real-time provides a foundation that can help writing researchers move towards SRL-driven writing scaffolds (Azevedo et al., 2011; Lajoie, 2005; Roscoe & Craig, 2022). We argue that by integrating established writing research methods (e.g., self-report measures, qualitative review of composed texts) with emerging data channels (e.g., keystroke logging behaviors), ongoing research can work to identify real-time instances of self-regulatory writing behaviors as students are composing and revising text, developing a process-level understanding of how students are self-regulating while writing. In doing so, continued research can build upon ongoing work in the SRL community to develop writing-specific SRL scaffolds that prompt first-year writing students towards enacting effective self-regulated writing behaviors during writing tasks.

By examining students’ reported self-regulated writing behaviors and their recorded keystroke behaviors, we hope to provide insight that can build upon writing-about-writing (WaW) curricula towards developing process-level self-regulated writing scaffolding. This is especially important as both writing curricula and intelligent writing systems (e.g., writing strategy tutoring systems; Roscoe et al., 2019) continue to prioritize a focus on the writing process rather than the final product alone.

Current study

We argue it is important that ongoing research on the impact of first-year writing programs looks beyond the outcome of self-regulated writing instruction (i.e., evaluating the quality of first-year writing students’ composed texts) to consider how students are (or are not) enacting self-regulated writing behaviors and how they perceive those behaviors while actively composing said texts. In doing so, researchers can work to better understand how self-regulatory writing behaviors unfold in real-time and the impact that self-regulated writing processes have on students’ ability to successfully engage with academic writing tasks, paving the way for first-year writing curricula capable of providing adaptive, process-level support.

For the current study, we are seeking to outline how we can develop scaffolding for college students’ self-regulated writing in a first-year writing course based on how students described their writing process in a self-reflective writing task as well as their keystroke activity while engaging in the self-reflective writing task. Note that we use the terms SRL strategies for writing and self-regulated writing synonymously because using SRL strategies for writing are components of self-regulated writing.

We posed the following research questions:

• 1. What is the distribution of students’ coded written responses to the self-reflective writing prompt based on phases of self-regulation?

• 2. Can we identify distinct student clusters based on their coded written responses to the self-reflective prompt?

• 3. Are there significant differences in keystroke activity (production, revision, pausing) between student clusters?

For this study, we do not pose hypotheses as the study is exploratory in nature since it was our first attempt to study self-regulated writing processes using our newly developed coding scheme. Addressing our research questions paves the way toward developing scaffolding geared towards groups of students who demonstrate needing assistance based on specific self-regulated writing processes.

Collecting students’ multimodal data channels provides in-depth information about their learning (specifically writing, in our study) processes (Azevedo & Gasevic, 2019). This can include, but is not limited to, log files or behavioral traces of student actions, such as mouse movement or keyboard entries. These events are all timestamped and provide the sequence of actions, making it comprehensive for data analysis (Azevedo & Taub, 2020). In addition, self-report measures provide information about students’ perceptions of their learning or factors that might impact it (Azevedo & Taub, 2020). However, although these data channels are continually used to measure self-regulated learning and writing in the literature, writing studies often do not rely on using process-based data channels directly produced from the writing process—i.e., keystroke data—without interference during writing, such as think-alouds (Bai, 2018; Hayes & Flower, 1980), interviews (Rose, 1980), or journaling (Roozen, 2009). There are several potential shortcomings when using only one data modality. Related to the current study, shortcomings to only using keystroke data can result in making assumptions about these input-based actions. For example, if the data is detecting instances of deleting and inserting text, this can suggest the student is rewriting content as a reflection strategy, but we don’t know if the student is truly reflecting without them telling us. Put differently, these data inform us of the student’s motor processes during writing, not their intentions or thoughts. Self-regulated learning is a (meta)cognitive process that is difficult to measure from behaviors alone (Winne & Azevedo, 2022). Additionally, shortcomings of only using self-report data include biases (e.g., experimenter bias) and being over- or under-confident (Hacker et al., 2009). Students often find it difficult to reliably report their feelings, thoughts, and actions, which is a major component of self-regulated learning.

As such, one of the major strengths of using multimodal data channels together for our study is that including multiple channels can account for information that is missing from only collecting one data channel. Collecting keystroke data alone will demonstrate how students were writing (e.g., producing or revising text), but without assessing how students describe their writing process, we would not be able to identify if students have a metacognitive awareness of themselves as writers (i.e., compare reporting writing with actual writing). While keystroke logging behavior alone cannot account for the multifaceted socio-cultural processes involved when students compose and revise texts (Graham, 2018), these kinds of analyses provide a complementary measure with which to compare students’ self-reported self-regulatory behaviors with their enacted keystroke behaviors as a means of generating a more holistic picture of students’ writing processes (both perceived and actual).

Methods

Participants and materials

Participants in this study were undergraduate students (n = 62) enrolled in a first-year writing course at a Southeastern university in the United States. Most participants were college freshman (87.1%) from a diverse range of academic majors (see Fig. 1) who reported a broad spectrum of feelings regarding their past (see Appendix, Table 2) and present experiences (see Appendix, Table 3) as writers. The course was designed to develop students’ writing skills through continued examination of their personal writing processes while introducing current methods for researching writing. All enrolled students were required to participate in this study as an assignment within the course, however students were free to withdraw from data collection at any point. Approval for this study was obtained from the university’s institutional review board.

Fig. 1

Distribution of reported academic majors (22 majors total).

Participants were asked to complete pre and post self-report questionnaires administered during the first meeting of the course and directly following the writing session. These questionnaires asked students to report on their motivation (AGQ-R; Elliot & Murayama, 2008), emotions (EV; Harley et al., 2015), and emotion regulation strategies (ERQ; Gross & John, 2003) as they relate to writing. We also included several researcher-developed items that asked students to report their feelings towards writing in high school. Table 4 (see Appendix) provides an overview of these questionnaires and associated sample questions, however participant responses to these items were not included in the current investigation.

We also collected students’ keystroke logging behaviors (see Sect. "Keystroke logging."), video recordings of students’ during the writing session, students’ facial expressions of emotions data, and students’ written reflections completed during the writing session. Only students’ keystroke-logging behaviors and written responses were included in the current analysis.

Self-reflective writing task

Students engaged in a scheduled 30-min writing session that took place in the writing lab. During the writing session, students were asked to respond to a self-reflective prompt by writing about their personal writing process (i.e., describe in as much detail the step-by-step process of how you complete a formal writing task). Participants were given additional suggested questions that they could address during the writing session (i.e., How do you start writing? How do you think of ideas to write about? How do you plan what to write? How do you manage time spent writing? How do you revise what you’ve written? How do you feel when you write? Can you also reflect on why some of these strategies do or don’t work for you, as well as how you learned them? What if you have to write with different tools?). These additional prompts were included to draw upon students’ prior knowledge and metacognitive awareness regarding their own self-regulation processes during writing (Negretti, 2012), however inclusion of these topics was optional. Students were explicitly told that their written responses would not be graded and were given no additional requirements (e.g., required length, structure, etc.) beyond the 30-min window to complete the writing session.

Experimental procedure

Students in this study participated in a one-day, 30-min session. During the first class-meeting of the semester (prior to the writing session), students were provided with a user ID and asked to complete the pre-test questionnaires via Qualtrics. Students then scheduled a time to complete the writing session in the writing lab later in the semester. Upon arriving at the lab for their scheduled writing session, students were provided with an overview of the study and asked to sign a consent form (students who completed the pre-test questionnaire but did not indicate their consent were excluded from this study). Students were then provided with instructions for the writing tasks before being given the writing prompt (students also received a hard copy of the prompt to reference as needed during the writing session). The researcher then calibrated the equipment by starting the video and keystroke recordings. Students then began the reflective writing task and were asked to continue writing for the duration of the 30-min session. After 30 min, the researcher stopped the recordings and instructed the students to complete the post-test questionnaires via Qualtrics. Students were then debriefed and thanked for their time before exiting the writing lab.

Data coding and scoring

For this study, we analyzed 2 types of data¹: (1) keystroke logging and (2) typed written responses to the self-reflective prompt. After collecting the data, we coded and scored each data type in preparation for our data analysis.

Keystroke logging

Keystroke logging behavior has been used to investigate a variety of real-time student writing behaviors such as cognitive processes while writing (Galbraith & Baaijen, 2019), differences in keystroke behaviors across distinct writing tasks (Conijn et al., 2019), using keystroke behavior to predict writing quality via final course grade (Conijn et al., 2022), and the impact of achievement motivation profiles on keystroke behaviors (Banzon et al., 2022). However, research using keystroke logging to examine SRL processes remains in its nascency (Bowen et al., 2022).

Students’ keystroke logging behaviors were recorded during the writing session using InputLog (Leijten & Van Waes, 2013). InputLog recorded each writing session by logging any form of input a student participant made in Microsoft Word (e.g., mouse movements or keystrokes) and an associated timestamp in milliseconds. InputLog then calculates the time elapsed between a writer’s inputs (i.e., the inter-key interval) to generate a fine-grain representation of how writers compose texts during a writing session, including the keystrokes student participants used to compose their text responses (see Fig. 2 for a visualization of keystroke behaviors). InputLog also logs cursor and character positions within the document, which allows the program to analyze revision behaviors by logging deleted text as well as new text that writers insert into previously composed text.

Fig. 2

Process graphs from InputLog showing two participants’ writing sessions. Left: participant using little to no revision, and constant keystroke activity. Right: participant using frequent revision, and long pauses to reflect or monitor.

We selected three types of keystroke activity behaviors outlined by Leijten and Van Waes (2013) with which to examine participants’ keystroke-logged data recorded via InputLog: producing, revising, and pausing. InputLog records that a writer is producing text when they are inputting text for the first time. We classify this as the production process of transcription described in Graham’s (2018) model. InputLog records that a writer is revising text according to two sub-behaviors: when they input new text into previously composed text (e.g., adding the letter ‘i’ within an existing “ths” to produce the word “this”), and when a writer deletes text using the backspace key. Based on Graham’s model, we classify revising as a combination of the reconceptualization and transcription production processes, and through the lens of self-regulated writing, we argue this behavior is part of the executive control function of reacting and the reconceptualization production process. InputLog records that a writer is pausing when they are not inputting any text. From a self-regulatory perspective, we argue that pausing could be indicative of multiple potential behaviors that are part of the executive control functions (i.e., planning or monitoring). A pause after composing a word may be indicative of a writer re-reading what they composed (i.e., monitoring/evaluating what they wrote) whereas a pause before composing new text may indicate that a writer is planning what they will write next (Van Waes et al., 2010). We used these three types of keystroke actions and the associated self-regulatory processes to examine participants’ self-regulatory behaviors during the writing session.

Keystrokes were scored by calculating the total time participants spent producing text (M = 392.19 s, SD = 177.49 s; range = 32.44–977.064 s), revising text (M = 200.67S, SD = 154.87 s; range = 20.62–685.48 s, and pausing (M = 1806.39 s, SD = 341.26; range = 1497.58–3936.12 s) during the 30-min writing session.

Coding written responses to the self-reflective writing prompt

Taking inspiration from Greene and Azevedo’s (2009) coding of think-aloud data that investigated both macro- and micro-level processes of self-regulated learning, we coded individual sentences composed by participants for both macro- and micro-level self-regulated writing processes. At the macro-level, students’ written responses to the reflective writing prompt were coded based on three stages of SRL (Greene & Azevedo, 2009; Zimmerman, 2013): planning, performance, and self-reflection. Each macro-level code contained several micro-level codes, which further specified the writing process(es) being described in the sentence (e.g., a sentence in which a participant described writing an outline would be coded as ‘planning’ at the macro-level and ‘outlining/making a written plan’ at the micro-level; see Sect. "Qualitative coding and analyses.", below). We only included macro-level processes in our quantitative analyses because we do not have sufficient statistical power to include all micro-level processes as separate variables. In addition, the frequencies of each micro-level process were too low, violating the required frequency per cell of a Chi-square analysis.

We used an iterative process to code students’ written responses, which consisted of three rounds of coding. Prior to coding, the researchers manually segmented the data by sentence. Each participant’s written text was broken up sentence by sentence using a period to delineate the ends of those sentences. These sentences were then coded by the research team. First, four research assistants coded the same subset of written sentences and used the first draft of the codebook to determine a macro- and micro-level code for each sentence. We did not randomly assign sentences to research assistants because we wanted to ensure they could gather contextual information when coding, i.e., they would be able to refer to the previous sentence(s) that might all relate to the same clause. For example, if a student wrote the following clauses as two separate sentences: “I wrote a lot in high school” and “That I did not enjoy doing”. We did this because if there was a sentence that included “that”, for example, we wanted coders to know what “that” meant by referring to previous sentences. After coding, the entire research team met to discuss disagreements and difficulty in coding in order to revise and finalize the codebook. Major sources of disagreements stemmed from the reflective nature of the task, such that it was at times difficult to distinguish between codes that were performance or reflection. For example, PB001 wrote “Unlike other people, my time management is not the best”. Initially, one research assistant coded this as macro code: Reflection and micro code: relating to others, while another research assistant coded it as macro code: Performance and micro code: flow of writing. This was because one clause in the sentence concerns how the participant manages time, but the sentence as a whole showed the participant reflecting on their process being different from others (i.e., shows how they reflect). This is why having the research assistants discuss and resolve disagreements was crucial for our study. Based on discussions of this initial round of coding, a fourth macro-level code ‘Other’ (and a subset of related micro-codes) was added to the coding scheme to account for sentences that fell outside of the three SRL categories.

In the second step, we divided participant data into three groups based on their enrolled course: Group A, Group B, and Group C. Two (out of the original four) research assistants were assigned to code sentences from Group A while the other two were assigned sentences from Group B. Research assistants from Group A compared their degree of coding agreement, initially reporting 71% agreement. Research assistants from Group B compared their degree of coding agreement, initially reporting 81% agreement.¹ Each pair of research assistants then discussed and resolved all coding incongruencies to reach 100% coding agreement.

In the third and final round of coding, one research assistant from each pair (i.e., one who coded sentences from Group A and one who coded sentences from Group B) was then assigned to code sentences from Group C. After initially reaching 74% agreement, the same process was repeated in order to reach 100% coding agreement.

Study analyses

For this study, we used a mixed-methods approach to examine students’ keystroke logging behavior (quantitative) and written responses of their self-reflections (qualitative). However, after coding all data, we assigned quantitative codes to the written reflections to conduct further analysis and address our research questions.

Qualitative coding and analyses

To determine which components of self-regulation students addressed during the writing task, we developed a coding scheme to code each written statement as one component of self-regulation: planning, performance, or self-reflection. These codes were developed loosely based on Zimmerman’s Socio-Cognitive Model of SRL (2000), as well as Greene and Azevedo’s macro- and micro-level SRL processes (2009). Greene and Azevedo (2009) outlined 4 macro-level (planning, monitoring, strategy use, and task difficulty and demands) and 35 micro-level SRL processes based on students’ verbalizations during learning with a computer-based learning environment.

We coded students’ written responses in 3 iterative steps (see Sect. "Coding written responses to the self-reflective writing prompt.", above). Our final codes included (see Appendix Table 5 for our codebook):

• (1) 4 macro-level codes: 3 macro level self-regulated writing codes (1 other, not pertaining to self-regulation). We report this category to include all codes, however this category is not relevant to our analyses and we therefore did not include it.

• (2) 18 micro-level codes: 5 for planning, 6 for performance, and 4 for self-reflection (3 other, such as structural/use of headings or re-typing the prompt).

Quantitative analysis

Once the qualitative data were coded, we transformed the data to generate proportion scores to account for frequency of codes in relation to the total amount students wrote (i.e., some participating students wrote more in their response than others). To calculate these, we used the formula:

$$\frac{frequency\;of\;code}{total\;number\;of\;codes}$$

We created a proportion score for each macro-level code (planning, performance, and self-reflection), for a total of 3 proportion scores per participating student. In calculating the total codes, we did include the ‘Other’ code to account for all codes; however, we did not calculate an ‘Other’ proportion score because it was not relevant to our research questions investigating self-regulatory processes. We ran a cluster analysis with the 3 student proportion scores:

• 1. Proportion of planning sentences (M_{planning-proportion} = 0.23, SD = 0.094)

• 2. Proportion of performance sentences (M_{performance-proportion} = 0.41, SD = 0.14)

• 3. Proportion of self-reflection sentences (M_{reflection-proportion} = 0.31, SD = 0.12)

Once we defined each cluster, we then used cluster membership as the independent variable for subsequent analyses comparing keystroke-logging behaviors between clusters. We performed a one-way MANOVA to compare production, revising, and pausing behaviors between the 4 clusters. Similar to the scores we created for the coded self-regulatory processes, we also created proportion scores for the keystroke logging actions in order to account for a range in keystroke activity. We used the formula:

$$\frac{time\;spent\;engaging\;in\;key\;stoke\;action}{total\;session\;time}$$

We created a proportion score for the three dependent variables:

• 1. Proportion of time spent producing text (M_{production-proportion} = 0.16, SD = 0.06).

• 2. Proportion of time spent revising text (M_{revision-proportion} = 0.082, SD = 0.06).

• 3. Proportion of time spent pausing during writing (M_{pausing-proportion} = 0.76, SD = 0.05).

These three proportion scores were the dependent variables for our MANOVA using cluster membership as our independent variable.

Results

3.1. Research Question 1: What is the distribution of students’ coded written responses to the self-reflective writing prompt based on phases of self-regulation?

To address this research question, we used our developed coding scheme (see Sect. "Qualitative coding and analyses.", above). We coded a total of 1,686 sentences (1,596 sentences excluding those coded as “Other”) generated by the student participants. Out of these sentences, students wrote about performance the most (684 sentences, 40.57%), followed by self-reflection (526 sentences, 31.2%), and then planning (386 sentences, 22.89%). Other was coded for 90 sentences (5.34%). A chi-square Goodness-of-Fit test (including the 3 macro-level codes only) indicated a significant value; χ²(2) = 83.56, p < 0.00001, indicating the distribution of self-regulated writing macro processes was not equal across all processes.

Sentences coded as performance involved writers discussing how they managed writing tasks during writing itself. This could have related to how they managed writing, such as how they maintained being in the “flow” of writing or managed their attention, how they generated ideas, used specific writing tools, or even revised. For example, participant PA002 wrote “I do a lot of revision a long [sic] the way with my writing” (micro code: revising), PA005a wrote “I just write till I'm finished” (micro code: flow of writing), and PA008 wrote “I write at my house in a comfortbale [sic] area where I do not feel distracted or stressed.” (micro code: flow of writing).

Sentences coded as self-reflection involved writers reflecting on and/or evaluating their writing processes. This could have involved expressions of their feelings or attitudes about their own writing, discussions about their cognitive processes during writing, how they compared themselves to other writers, or even how they discussed their own development as a writer. For example, participant PC018 wrote “I never know how to start or how to put all my ideas into words,” (micro code: thought/writing processes). PB003 wrote “However, when I am writing about something I find intreguing [sic] or something I am passionate about my writing process seems to flow easier,” (micro code: thought/writing processes) and PB010 wrote “If I am writing something creative, I might feel more happy and energetic to write.” (micro code: feelings/attitudes about own writing).

Sentences coded as planning involved writers discussing how they planned (or didn’t plan) for writing tasks. This could have involved writers discussing how they prepare for writing tasks, how they activated prior knowledge, how they determined which strategies to use, how they addressed motivations for writing, as well as how they used outlines or other written plans. For example, participant PA004 wrote “Sometimes I will write out an outline if it is mandatory, but if it is not I find myself hopping right into the paper,” (micro code: outlining/making a written plan), PA010 wrote “My outlines typically consist of bulletpoints answering prompt questions and topics I want to write include,” (micro code: outlining/making a written plan), and PC021 wrote “I either look up an idea on the internet that are based off previous knowledge or If I want to go more in dept with a research question that I think it's interestingy [sic]” (micro code: prior knowledge activation).

3.2. Research Question 2: Can we identify distinct student clusters based on their coded written responses to the self-reflective prompt?

For this research question, we ran two types of cluster analysis: hierarchical clustering followed by k-means clustering, to determine if we could group students based on their coded written responses (as described in Research Question 1).

Prior to the analysis, we standardized all scores, and then used several indicators (as outlined in Wortha et al., 2019) to determine the ideal number of clusters. First, we applied hierarchical clustering on the standardized scores using Ward’s method to compare change in agglomeration coefficients (ACs). The ACs indicated a 2- to 5- cluster solution, with changes in AC from 7.048 for a 6-cluster solution to changes in 13.134, 14.77, 20.587, and 34.395 for 5-, 4-, 3-, and 2-cluster solutions, respectively. However, with the 5-cluster solution, there were n < 10 cases for at least one cluster; therefore, we ruled out a 5-cluster solution. We then used k-means clustering with 2, 3, and 4 clusters. To justify our choice of cluster solution, we compared our possible cluster solutions on 4 parameters: the iteration history, ANOVA table comparing means between standardized values of all variables used in the cluster analysis, differences in cluster centroid values, and sample size per cluster (Hair et al., 2010). First, all cluster solutions’ iteration histories reached 0.00, indicating clusters were created without changing the centroid values. For the 2-cluster solution, cases were very imbalanced (n = 22 vs. 40), and based on the ANOVA table, there were not significant differences between the means of each cluster. We therefore ruled out a 2-cluster solution. We compared cluster centroids between the 3- and 4-cluster solutions. ANOVA tables for both 3- and 4-cluster solutions demonstrated significant differences between the cluster means; however, the largest cluster centroid differences were for the 4-cluster solution. In addition, the 4-cluster solution used an iteration history of 5, while the 3-cluster solution used an iteration history of 9. The group distribution was the most balanced for the 4-cluster solution. Therefore, we decided on using a 4-cluster solution for further analyses (see Fig. 3).

Fig. 3

Final cluster centers for 4-cluster solution. Note. Values = SD, as we used standardized scores.

Using the 4-cluster solution, we named the following 4 clusters:

• 1) Planners, n = 20

• 2) Anti-Planners, n = 14

• 3) Performers, n = 13

• 4) Reflectors, n = 15

Cluster 1: Planners

The first cluster contains the highest proportion scores (1SD above the mean) for sentences that discussed planning, with proportion scores for sentences about performance (-0.64SD) and self-reflection (-0.33SD) below the mean (performance scores were the lowest). In fact, this was the only cluster to have proportion scores for planning sentences above the mean. From these data only (i.e., not their keystroke activity), we therefore presume these students are effective at engaging in the forethought phase of self-regulation with less focus on the later stages. As a result, we called this cluster The Planners.

Cluster 2: Anti-planners

In contrast to the planners, this cluster demonstrated proportion scores higher than the mean for sentences about performance (0.54SD) and self-reflection (0.41SD), with proportion scores for sentences about planning -1.07SD below the mean. Although these students do not demonstrate the highest proportion scores for sentences about performance and self-reflection, this was the only cluster that contained proportion scores for 2 coded writing processes that were above the mean; and they demonstrated the lowest score for proportion of sentences about planning. As such, we called this cluster The Anti-Planners.

Cluster 3: Performers

This cluster contained the highest proportion score for sentences that discussed performance (1.23SD). Students in this cluster also had the lowest proportion scores for sentences about self-reflection (-1.26SD, lowest score out of all variables across all clusters), as well as proportion scores for planning below the mean (-0.14SD). Given this, we might assume, from the coded written data only, these students are not strong self-regulators because they focus on performance, and not planning or reflection. However, as we cannot confirm this from only one data channel, we called this cluster The Performers.

Cluster 4: Reflectors

This final cluster is characterized by the highest (1.14SD) proportion score of sentences about self-reflection and the lowest (-0.71SD) proportion scores on sentences about performance. This cluster also had low proportion scores on sentences about planning (-0.21SD), although not the lowest of all clusters. Based on the reflection behavior noted in this cluster, it does seem these students are stronger at engaging in a later phase of self-regulation. Given this, we named this cluster The Reflectors.

In general, based on these findings, we were able to create distinct clusters of students based on how they discussed self-regulatory processes as they described their writing process. This demonstrates their description of planning, performance, and self-reflection phrases and, more importantly, they do possess the self-regulatory skills needed to engage in self-regulated writing.

3.3. Research Question 3: Are there significant differences in keystroke activity (production, revision, pausing) between student clusters?

To address this research question, we ran a one-way multivariate analysis of variance (MANOVA) with cluster membership as our independent variable. The three dependent variables are the proportions of keystroke logging variables: proportion of production, proportion of revision, and proportion of pausing (see coding and scoring, Sect. "Quantitative analysis" above for details). The Levene’s test of equality of error variances were not significant for any of the dependent variables (all p > 0.05), confirming the assumption of equal variances was met.

Results revealed a significant MANOVA; Wilks’ λ = 0.78, F(6,114) = 2.55, p = 0.024, η_p² = 0.12, observed power = 0.83. Tests of between-subjects effects (see Table 1) revealed a significant effect for proportion of production; F(3, 58) = 3.24, p = 0.028, η_p² = 0.14, observed power = 0.72 and proportion of revision; F(3, 58) = 2.89, p = 0.043, η_p² = 0.13, observed power = 0.66, but not for proportion of pausing; F(3, 58) = 1.85, p = 0.15, η_p² = 0.09, observed power = 0.46.

Table 1 Descriptive Statistics and Between-Subjects Effects Comparing Keystroke Data by Cluster

Post-hoc analysis using Tukey’s HSD (see Table 1 (right), Fig. 4) revealed there are significant differences in proportion of production between clusters 2 and 3 (anti-planners vs. performers) and proportion of revision between clusters 1 and 2 (planners vs. anti-planners). More specifically, students in cluster 2 (anti-planners) had significantly larger proportions of engaging in production behavior compared to students in cluster 3 (performers), and students in cluster 1 (planners) had significantly larger proportions of engaging in revision behavior compared to students in cluster 2 (anti-planners). Therefore, it appears students in cluster 3 (performers) demonstrated smaller proportions of engaging in both production and revision keystroke behaviors compared to other clusters.

Fig. 4

Mean comparisons of production and revision proportions between clusters. *p< .05 for mean difference. Note. Pausing proportion was not included because there were no significant differences between clusters in this keystroke behavior.

Discussion

The goal of this study was to investigate how students reported engaging in self-regulated writing strategies during a writing task, and how that related to their keystroke-logging behavior—towards determining how we can best scaffold these students in using specific self-regulated processes during writing. Findings from our study included: (1) qualitatively coding and scoring our data, (2) quantifying the data to outline different categories of self-regulated writing strategies and distinct clusters of students demonstrating different use of those strategies, and (3) examining behavioral differences between those clusters of students.

Overview of findings

Research Question 1: What is the distribution of students’ coded written responses to the self-reflective writing prompt based on phases of self-regulation?

Our first research question allowed us to propose and outline a new coding scheme for self-regulated writing processes based on an existing, well-established coding scheme (Greene & Azevedo, 2009) for macro- and micro-level self-regulated learning processes that was developed based on coding think-aloud data. Our coding scheme details 3 macro-level (and 1 “other”) and 18 micro-level self-regulated writing processes. After coding the data, descriptive statistics highlighted students did write about these processes during the writing task, but at different frequencies. This indicates students entering first-year writing courses do have an awareness of engaging in these writing strategies, even though at times, some students do not believe they are good writers (Bruning et al., 2013; Downs & Wardle, 2007). This is an important result with implications for building individualized support in the classroom (see Sect. "Considerations for the classroom.", below) because students may not need support in all aspects of self-regulatory skills. Contrary to a common argument that self-regulation skills are either present or not (e.g., state vs. trait based; Winne & Perry, 2000), our results demonstrate students can possess some regulatory skills, with different levels of those skills (e.g., some clusters had higher planning while others had lower or below average planning). Therefore, individualized support should focus on which elements of self-regulation students are demonstrating needing help with based on their self-reflections about writing (see Sect. "Towards Developing Scaffolding to Support Self-Regulated Writing" for additional discussion). In addition, our results demonstrate how self-regulated writing does not look the same for everyone, as seen from the coded written responses. Our codebook (Appendix Table 5) contains many examples of what each macro-level process looks like by having several micro-level processes associated with each one. Therefore, even if students within one cluster all have higher than average proportions of planning, for example, the actual sentences discussing planning could look very different across students (e.g., using outlining vs. brainstorming). We believe this is why individualized support is crucial—it focuses on how self-regulatory processes are used for each student.

Research Question 2: Can we identify distinct student clusters based on their coded written responses to the self-reflective prompt?

Addressing this research question included using cluster analysis (hierarchical and k-means clustering) to determine if we could define distinct clusters of students based on their coded written responses to the writing task. Using z-scores of the proportions of planning, performance, and self-reflection codes, we distinguished between 4 clusters of students: (1) planners, (2) anti-planners, (3) performers, and (4) reflectors.

In comparing all 4 clusters, three of the clusters (planners, performers, and reflectors) demonstrated a higher than average cluster center for one self-regulated writing process. The fourth cluster, anti-planners, showed a much lower than average cluster center for planning, but did obtain higher than average cluster centers for the other 2 processes (performance and self-reflection). Therefore, even though we do not see a “dominant” self-regulated writing process for the anti-planners, this is the only student cluster that demonstrated higher than average cluster centers for 2 processes. This leads to the question—does this imply the anti-planners are the best self-regulators, are they sufficient self-regulators, or are they simply worse at discussing planning?

It would be typical to assume a “good” self-regulator would balance their discussion by describing all components of self-regulation—i.e., similar cluster centers across all 3 self-regulatory processes would necessitate a “self-regulators” cluster. However, none of the clusters demonstrated this behavior of having high proportions of each self-regulatory process. Based on this finding, we do not believe students are demonstrating “bad” self-regulatory behaviors or are “bad” self-regulators. Rather, these results have implications for designing scaffolds that target particular components of self-regulation (see Sect. "Towards Developing Scaffolding to Support Self-Regulated Writing", below) because all clusters do demonstrate students wrote about some elements of self-regulation. Therefore, our findings demonstrate all students did report self-regulating to some extent, but there are particular self-regulated writing strategies some groups discussed more than others. It seems like scaffolding should include all components of self-regulation in general, as opposed to only one component, because no student cluster similarly reported on their use of self-regulated writing processes. However, if we think about more individualized or adaptive scaffolding, it seems like each of these clusters would benefit from scaffolding that focuses on particular (and different) components of SRL, because these clusters demonstrate all students do discuss some element of regulation already.

Research Question 3: Are there significant differences in keystroke activity (production, revision, pausing) between student clusters?

This third research question used a one-way MANOVA to determine if there were differences in keystroke-logging behaviors between the four identified clusters from research question 2. Results demonstrated a significant test result, such that participating students across some clusters engaged in different keystroke behaviors indicative of self-regulated writing processes. Specifically, students in the anti-planners cluster (who generated below average proportions of planning sentences and above average proportions of performance and self-reflection sentences) engaged in significantly higher proportions of production keystrokes than students in the performers cluster, and significantly lower proportions of revision keystrokes than students in the planners cluster.

These findings are particularly important because it demonstrates a potential disconnect between using multiple data channels, i.e., self-report vs. keystroke logging. Based on the result of significantly more revision, we would assume these students engage in more self-regulation during writing because revision includes deleting and inserting text, which requires making adaptations and self-reflective self-regulatory processes (Winne & Hadwin, 1998, 2008). When interpreting our clusters, we presumed the anti-planners were stronger self-regulators because they had higher than average cluster center means on 2 self-regulatory processes, as opposed to the other 3 clusters who only demonstrated above average cluster centers for 1 self-regulatory process. However, based on the keystroke behaviors, the planners were the ones who engaged in more revision behaviors than the anti-planners (and no significant result including the revisors) even though they had lower than average cluster centers for performance and self-reflection.

In addition, the anti-planners engaged in more production behaviors than the performers, which suggests they focused more on discussing performance than planning or self-reflection; not just not on planning, as suggested by their cluster name. We would assume the performers would engage in more production behavior because this keystroke activity demonstrates more performance than pausing or revising, which we view as planning or self-reflection behaviors. This poses an important consideration that must be made when using multichannel data and considering which channels to “trust”. As discussed in the Introduction, research has demonstrated self-report data can sometimes be unreliable (Azevedo & Taub, 2020). For example, students do not accurately report their understanding of material (i.e., inaccurate metacognitive judgment; Hacker et al., 2009; Taub et al., 2021). In this case, perhaps students may not accurately report their use of self-regulatory processes during writing. Further, cognitive models of writing (Galbraith, 1999; Graham, 2018; Hayes, 2012) suggest that writing involves accessing long-term or episodic memory as part of recursively translating, evaluating, and transcribing ideas onto paper. Therefore, how students recalled and reflected on their writing processes prior and during writing might have affected their writing during the task. This could explain why the anti-planners engaged in more production than the performers, as they may have avoided much of any planning or reflection while writing and instead prioritized writing itself. Although we do not have sufficient data to support these claims, this disconnect does pose important implications for using multichannel data and choosing which data channel to “trust” when there is that disconnect. Despite this uncertainty, results do, in fact, suggest we can detect instances of self-regulated writing from keystroke and self-report written responses, demonstrating that students do engage in self-regulation during writing, it is just less clear if and how the keystroke data are related students’ self-reported using of self-regulated writing strategies.

Limitations and their future directions

While our study does yield interesting findings with important implications towards developing self-regulated writing scaffolding (see Sect. "Towards Developing Scaffolding to Support Self-Regulated Writing"), we must mention the limitations and how we aim to address them in future studies.

First, our coding process included assigning only 1 code to each statement (i.e., assign one self-regulated writing process to each sentence). It is possible students did not distinguish between their writing processes while engaging in this writing task. Therefore, perhaps students engage in more self-regulatory processes compared to what they reported, but coders were limited to selecting one code only. In future studies, we will employ a co-occurring codes option for coding, such that phrases will be able to be coded by more than one single macro-level or micro-level code, as appropriate.

Another limitation pertains to our new coding scheme. We are excited to present this new method of coding written responses from a self-regulatory perspective based on theories of SRL, however we still need to validate it to confirm if this coding scheme would be appropriate for coding writing beyond our study. Future studies will aim at validating the coding scheme to determine if it can be used for coding other writing prompts, or if it will only pertain to the prompt we used for this study. Additionally, as the coding scheme is based on Greene and Azevedo’s (2009) coding of think-aloud data, future studies should also aim at using our coding scheme for think-aloud and interview data collected during writing tasks.

A third limitation pertains to the use of self-report data. Students’ written responses were essentially them self-reporting their writing strategies, which we then coded into phases of self-regulation. We cannot be sure if what students reported doing while they write is truly how they engage in the writing process. We attempted to address this issue by analyzing data from multiple data sources (i.e., keystroke logging and written responses), but since there is no guarantee responses reflected how students habitually write, future studies should attempt to form a one-to-one match between self-reported and actual writing behaviors.

Lastly, related to the use of self-report measures, an important limitation relates to how the task we used is a reflective task itself. Therefore, we were asking students to engage in self-regulation by reflecting on their writing process. This means we were likely going to detect instances of self-regulated writing due to the nature of the writing prompt. It is less clear, therefore, if students typically reflect on their writing when engaging in writing tasks different from the one used for this study. The nature of the writing task made it difficult to distinguish between coding for self-reflection or other self-regulatory processes. in other words, all written statements consisted of them reflecting, making it hard, at times, to tease apart statements that were self-reflective in general (e.g., reflecting on planning or performance) compared to students reporting they engage in reflection during writing. This further demonstrates the need for future studies to require students to engage in a writing task that is not self-reflective in nature already (e.g., summarize a book chapter, write a lab report).

Towards developing scaffolding to support self-regulated writing

One of the main goals of our study is to highlight how we can develop tools to scaffold students during writing to ensure they: (1) have the skills to engage in effective writing processes (i.e., self-regulated writing strategies), (2) develop an awareness of their abilities as writers, and (3) sustain the motivation to use these self-regulated writing processes in their academic as well as future careers—as all careers do require some component of writing. As such, we believe implications from this study pave the way towards what we should be scaffolding, who we should provide scaffolds to, and how these scaffolds should be provided.

In developing scaffolding and support for students, we believe there are two types of considerations that need to be made based on interpreting the results from our study: (1) considerations for the classroom and for interpreting data, and (2) considerations for developing real-time interventions. These considerations are not necessarily mutually exclusive; however, we believe categorizing them will be helpful for future researchers and educators studying and teaching self-regulation and writing in different contexts.

Considerations for the classroom

When designing scaffolded support for writing to students in the classroom, we believe our findings suggest this support should start with the teacher first understanding who the students in the classroom are. For example, are these students taking their first ever writing course? Are these students freshmen? We advise teachers to adapt their curriculum based on these factors because we cannot expect students to be proficient at self-regulated writing if they have never been introduced to these constructs before. If students are novice writers, depending on their backgrounds, they might not “know” themselves as writers. They might have written what they were expected to do in their writing classes, not what they actually do. As such, it is important for teachers to remember that they might be working with writers with different levels of expertise based on their students’ backgrounds. In our study, we did ask students about the support they received regarding planning and revision in high school, which we believe could play an influential role in how they self-regulate in college. We will examine the influence of prior self-regulatory support in future studies.

Moreover, we believe teachers need to link self-regulation with writing behaviors to fully understand how students engage in writing prior to providing them the support they need. For example, does the student frequently engage in planning processes? As such, the student would not need support that focuses on planning, but rather needs support to focus on other stages of self-regulation such as performance or self-reflection.

However, we also need to differentiate between what students think and what they actually do in terms of their writing. Do students report their use of self-reflection, but their keystroke behaviors suggest otherwise (similar to what we found in our results)? If we can pinpoint this disconnect, it would be crucial for teachers to focus on ensuring students understand themselves as writers, in addition to the teachers understanding students’ writing processes as well. As such, based on our study, if we know there are different groups of students who write about how they engage in different self-regulatory processes more than others, and we found some differences in how the groups of students engage in keystroke-level activities, it will be important to support students in the classroom based on their specific writing needs. The specific writing needs will depend on the student (i.e., foster self-reflection for students who do not reflect) and the task itself (i.e., different tasks require different strategies). For example, as demonstrated by Galbraith and Torrance (2004), planning is particularly crucial for successful writing, but different writing tasks may benefit from different kinds of planning. As such, we do not necessarily recommend equal engagement in all self-regulatory processes, But future studies may yield further insight.

It seems that even though students do not discuss all aspects of self-regulation, their keystrokes demonstrate they do engage in self-regulation. As we move toward designing scaffolded support to students, we must make sure to consider which types of data will accurately present what students are doing compared to what they think, if the 2 are not aligned. Conversely, if we want to provide scaffolding based on potential misconceptions students have about their use of self-regulatory processes, it could be beneficial to present them with conflicting data channels to raise their awareness of themselves as writers, which is one of the goals of the writing-about-writing curriculum (Downs & Wardle, 2007). Additionally, having students reflect on their process-based reports (see Vandermeulen et al., 2020a, b) in conjunction with their self-reports of their writing processes may allow them to independently note these conflictions and make alterations to their writing (Vandermeulen et al., 2023) Therefore, based on our findings from this study, we suggest for implementing scaffolded support based on collecting student data (e.g., show students their progress graph and have them reflect on it). It is important to understand what the data are informing us about the students and what the data are informing students about themselves.

Considerations for developing real-time interventions

Another important consideration that should be made when developing scaffolded support for students is when and how the support will be provided to students. The previous considerations discussed the who and what components of providing support. This consideration addresses the timing of the support—should it be based on student events or based on particular timepoints during a writing session? If it were event-based, the support would be more individualized because the scaffold would be provided based on specific actions the student is doing. For example, if a student has not engaged in any planning behaviors, the student can be prompted to do so. In contrast, action-based scaffolds occur at a given point in time for everyone. For example, all students are prompted to plan after 10 min into a writing session. Based on our findings, and based on the considerations we discussed regarding data alignment, we believe it could be useful to use event-based scaffolding that includes time-based prompting as well. For example, the four student clusters demonstrate there are groups of students who could each benefit from discussing specific components of self-regulation, such as cluster 3 (performers), who had very low proportions of discussing self-reflection and below average (not the lowest) proportions of discussing planning. However, the clusters are not indicating students do not discuss these processes at all. Perhaps, for example, some of these students mention planning or self-reflection at the beginning of the session, but then stop doing so. In this case, perhaps these students would benefit from scaffolds about certain processes after a certain time limit of not discussing them.

Future directions

In addition to the future directions that aim to address our study limitations, we propose additional future directions that will address other factors that have been found to impact self-regulation, such as emotions and motivation (Schunk & DiBenedetto, 2020; Taub et al., 2021).

First, research demonstrates the impact of motivation on learning and performance across learning environments (Schunk & DiBenedetto, 2021), such that when students have high levels of motivation, this can lead to greater learning outcomes. Among many motivational constructs is self-efficacy, defined as how well one believes they are at an activity or their ability to complete a task (Schunk & DiBenedetto, 2021). In our study, it is possible that writers who feel good about themselves as writers (high writing self-efficacy) are not necessarily good self-regulators, i.e., they know what they do and understand why, but maybe just because of their experiences; they do not feel a need to develop self-regulatory skills. Future studies should examine how writing self-efficacy, and other motivational factors such as achievement goals (via the AGQ) play a role in students’ writing processes. As indicated in the Appendix, we did collect these data for future studies.

Research also demonstrates the impact emotions have on learning (Pekrun & Linnenbrink-Garcia, 2014). In our study, we collected videos of students facial expressions, but we did not include this in the current paper. It is possible student emotions could impact or either be impacted by their writing processes. For example, perhaps a student expresses anger after a long pause. It is possible the student was trying to get their writing started and were planning how to start, but was hitting a mind-block and could not decide how to start, causing feelings of anger or frustration. In contrast, the student might have felt frustrated about what they had just written (maybe it was difficult to get their words expressed), leading to a long pause where they re-read what they wrote. Future studies will incorporate student facial expressions with time-series analyses to investigate student emotions and the temporality of which came first—the emotion or writing behavior.

Future studies will also incorporate additional data, such as interviews so we can discuss findings from the data with participants to gain ground truth about what the data are telling us. As mentioned earlier, there are sometimes conflicting interpretations from results using either the same or different data channels—i.e., clusters vs. keystroke data, temporal sequence of emotions and keystroke behavior. By asking students directly, we can confirm what exactly they were thinking during writing. We can also address questions about whether students choose to focus on only one self-regulated writing process, as suggested in their written responses, or if they engage in self-regulated writing processes simultaneously, but only chose to report one process (and why they did so). Our coding scheme also did not include an assessment of the effectiveness of the strategies students wrote about. For example, students discussed their planning strategies, which we coded as planning. It is possible these planning strategies are not the most effective (e.g., writing an outline vs. verbally brainstorming—we argue written plans are more effective so students have a record of these plans). It was not our intention in this study to evaluate the effectiveness of self-regulated writing strategies, however this would be a valuable area for future research.

In addition to the data channels we used for the current study, there are many additional data channels with which we can investigate SRL in discipline specific areas with increased granularity. Many studies have used eye tracking (e.g., Taub et al., 2017; van Marlen et al., 2022), facial expressions of emotional states (e.g., Dever et al., 2022; Sümer et al., 2021), think-aloud protocols (e.g., Bai, 2018; Greene & Azevedo, 2009; Greene et al., 2018), log files, learning analytics, or clickstream data (e.g., Cogliano et al., 2022; Taub et al., 2022), and other types of data, such as reflective journals (Negretti, 2012, 2021) to investigate SRL processes during learning. Therefore, in future studies, we aim to include additional data channels that help further describe the relationship between self-regulated writing and the cognitive, metacognitive, emotional, motivational, and social factors that impact it.

Lastly, future studies will focus on the temporal nature of self-regulation, and will include micro-level processes. As mentioned, we only discussed the 3 macro-level self-regulated writing processes (planning, performance, and self-reflection) in this paper, and used those to conduct our cluster analysis. This was done to simplify the analysis, however if we use analyses such as hierarchical linear modeling and additional educational data mining techniques such as sequence mining, we aim to address the finer-grained coded self-regulated writing processes. For example, we can use sequential pattern mining to investigate student patterns in micro-level processes or keystroke behavior during a writing session, and then use differential sequence mining to compare the frequencies of patterns between groups, such as the 4 clusters determined in the current study, students with low vs. high writing self-efficacy, etc. Engaging in all of these future studies will advance our research using multimodal data to examine self-regulated writing in college.

Conclusion

The goal of this paper was to highlight our approach to investigating college students’ writing processes as they responded to a written self-reflective prompt. We combined theoretical frameworks of self-regulated learning and writing to create a coding scheme that assigns macro- and micro-level codes to each written sentence. Using the frequency of assigned codes, we created proportion scores of each macro-level code, which we then standardized and used as input variables for conducting a cluster analysis to determine if we could outline groups of student writers based on the amount of self-regulatory processes they discussed using in their written reflections. Once we discovered four student clusters, we compared their keystroke-logging behavior using a one-way MANOVA, with post-hoc analyses revealing significant differences between production and revision behavior between some clusters, but no differences in pausing behavior. Results have important implications for the design of scaffolding that targets students’ self-regulated writing processes, such that scaffolded support should focus on particular phases of self-regulation for different groups of students. We also provide recommendations on making scaffolding design decisions, such that we need to understand: (1) who students are in the classroom, (2) how to interpret their behaviors if using multimodal data, (3) when to provide support, and (4) how scaffolding should be given (i.e., event- or action-based, or both). Future studies are needed to test the most beneficial ways to scaffold students to ensure they are engaging in effective writing strategies that promote higher levels of metacognitive awareness of one’s writing to ensure students are effective writers throughout their years in college and beyond.

Appendix

Tables 2, 3, 4 and 5

Table 2 Students’ Reported Feelings about their Writing Experiences in High School

Table 3 Students’ Reported Emotions When Starting a New Writing Task for School or Work

Table 4 List of Questionnaires Completed for the Study

Table 5 Codebook for Macro- and Micro-Level Self-Regulated Writing Processes