Introduction

Promoting critical thinking is a primary goal in higher education targeted for future knowledge workers in the United States (Bennett et al. 1999; Brookfield 1987; Hagedorn et al. 1999; Harvey et al. 1997). Critical thinking is a higher-order thinking skill associated with the ability to think reasonably and reflectively and decide what to believe or perform (Ennis 1985; Glaser 1985; Kuhn 1999; Paul 1990). Paul and Elder (2001) defined critical thinking as “the intellectually disciplined process of actively conceptualizing, applying, analyzing, synthesizing, and evaluating information gathered from observation, experience, reflection, reasoning or communication as a guide to belief and action” (p. 371). As the art of thinking about an individual’s thinking, i.e., a meta-cognitive skill (Beyer 1995; Sharma and Hannafin 2004), critical thinking has been explained in many different ways such as critical inquiry, problem solving, and cognitive presence (Angeli et al. 2003; Angelo 1995; Garrison et al. 2000).

In order to foster critical thinking, computer-supported collaborative learning’s (CSCL) pedagogical benefits have been considered since the mid-1990s (Henri 1992; Land and Dornisch 2001; Newman et al. 1995). Particularly within online discussion forums, researchers have tried to find effective instructional strategies to improve critical thinking skills (Anderson et al. 2001; Angeli et al. 2003; Angelo 1995; Collison et al. 2000; Duffy et al. 1998; Garrison et al. 2000). Scaffolding is considered one of the most important strategies for developing higher-order thinking in online collaborative learning environments (Ahern et al. 1992; Salmon 2000; Salomon et al. 1989; Zhu 1998).

Scaffolding is instructional assistance, such as guides, strategies, and tools, that helps learners achieve a higher level of development than would be possible without the assistance (Brush and Saye 2002; Hannafin et al. 1999; Hogan and Pressley 1997; Linn 1995; Vygotsky 1978; Wood et al. 1976). Appropriate implementation of scaffolding helps learners gradually internalize relevant principles and take independent responsibility to achieve a higher developmental level (Vygotsky 1978). The concept of scaffolding was initially considered as personal in nature, i.e., tutoring or additional assistance provided by teachers or peers to facilitate the instructional process in a learning setting (Wood et al. 1976). Since the emergence of computer technologies in education, however, scaffolding has included tools and resources embedded within computer software to support students in instructional activities (Brush and Saye 2002). In terms of categorizing scaffolds in computer-based instruction, there are three classification systems (Tan 2005): (a) delivery control mechanisms, (b) functions, and (c) nature.

In terms of the delivery control mechanism of scaffolds in computer systems, Guzdial (1994) divided scaffolds into adaptable scaffolding that learners can change or move and adaptive scaffolding that computer systems control. Regarding the functions of scaffolds, Jackson et al. defined three types of scaffolds in teaching software: (a) supportive scaffolds support tasks by guiding, coaching and modeling learning processes, (b) reflective scaffolds help learners plan, predict and evaluate their tasks, and (c) intrinsic scaffolds change the difficulty of a task. Hannafin et al. (1999) created four additional categories of scaffolds for the Open Learning Environments where learners can design their instructional goals and activities. Procedural scaffolds guide how to utilize tools. Conceptual scaffolds help to know what is under consideration. Strategic scaffolds guide how to approach problems, and meta-cognitive scaffolds guide how to think in learning. Based on the nature of scaffolds, Brush and Saye (2002) conceptualized two types of scaffolds: (a) soft scaffolds are dynamic and situation-specific aid provided by instructors or peers, and (b) hard scaffolds are static supports planned in advance.

Scaffolding to enhance critical thinking in face-to-face learning situations includes modeling, externalizing reflection, meta-cognition, and Socratic questioning (Beyer 1995; Sharma and Hannafin 2004). However, in online collaborative learning environments, additional pedagogical benefits and challenges should be considered. McLoughlin and Luca (2000) emphasized two features of CSCL in terms of scaffolding critical thinking: (a) the interaction of instructors offering questions and feedback and (b) the role of computer-mediated communication tools providing facilitating dialogue, cooperative learning, and the presentation of multiple viewpoints. Focusing on these scaffolding features, various strategies have been developed to promote critical thinking skills, especially for online discussion forums. Table 1 shows four sets of scaffolding tactics and examples designed for text-based communication conferencing systems in asynchronous message transmission (Anderson et al. 2001; Angeli et al. 2003; Angelo 1995; Collison et al. 2000; Duffy et al. 1998; Garrison et al. 2000).

Table 1 Scaffolding strategies to improve critical thinking in online discussion forums
Full size table

Collison et al. (2000) proposed scaffolding strategies to aid the internal process of an individual learner during critical thinking. Their main argument is that scaffolding for enhancing critical thinking should be integrated in the process of (a) sharpening the focus and (b) deepening the dialogue. Scaffolding for sharpening the focus facilitates learners to clarify their ideas, bringing a common understanding to learners. Also, this type of scaffolding strategy can highlight relevant ideas and key contributions, bring coherence, and push the dialogue forward. As for strategies for deepening the dialogue, learners’ thoughts can be extended through more in-depth discussions on common ground, where learners examine their own beliefs and assumptions and reflect on perturbations to build new levels of understanding.

In order to observe students’ critical thinking in class activities, Brookfield (1987) created nine indicators for practitioners as follows: (a) distinguishing between verifiable facts and value claims, (b) distinguishing relevant from irrelevant information, claims, or reasons, (c) determining the factual accuracy of a statement, (d) determining the credibility of a source, (e) identifying ambiguous claims or arguments, (f) identifying unstated assumptions, (g) detecting bias, (h) identifying logical fallacies, and (i) recognizing logical inconsistencies in a line of reasoning.

Greenlaw and DeLoach (2003) developed a theoretical framework to examine college students’ critical thinking levels in online discussion: (a) Level 0—off-the-subject or otherwise unscorable, (b) Level 1 (unilateral descriptions)—paraphrasing information, repeating or restating the question, (c) Level 2 (simplistic alternatives/argument)—taking a side, not exploring other alternatives, making unsupported assertions, or making simplistic arguments, (d) Level 3 (basic analysis/reasoning)—making a serious attempt to analyze an argument or competing arguments or competing arguments and evaluate it/them with evidence, (e) Level 4 (theoretical inference)—employing the use of theories to make a cohesive statement, (f) Level 5 (empirical inference)—adding to the level of sophistication by introducing empirical evidence to strengthen their theoretical argument, and (g) Level 6 (merging values with analysis)—moving beyond objective analysis to incorporate subjective interests. Greenlaw and DeLoach’s (2003) framework can be a practical guideline for practitioners who want to promote and measure the development of critical thinking skills in actual online collaborative learning situations. Their framework is well organized according to the hierarchy of increasingly sophisticated critical thinking, using the criteria for argumentation and evidence (Osman 2008), which allows for the analysis of the observable development of critical thinking in online collaborative learning environments such as wikis.

Wikis such as Wikibooks and Wikipedia have been suggested as innovative scaffolding tools that educators should adopt for new generation of learners in the Web 2.0 era (Bonk 2009). Wikis are knowledge-building communities that maximize collective intelligence through dynamic knowledge-sharing and creating using web technologies (Högg et al. 2006). Wikibooks are a free collection of open-content textbooks that multiple users can develop and edit on the Internet (Bell 2009) as part of the Wikimedia Foundation’s services along with Wikipedia. A wikibook has a book title and a table of contents including chapters. Each chapter page has a chapter title and four page tabs on top, named Read (i.e., wikibook module), Edit, View History, and Discussion. To modify the wikibook, users must enter an Edit mode and save a new version of the page; this action creates a chronological structure on History pages (Brodahl et al. 2011). The Edit mode allows users to revisit, edit, and review the pages previously written by another user.

In formal learning, this attribute of collaborative writing can enhance the quantity and quality of interactions among learners (Xiao and Lucking 2008) as well as improve their motivation and participation (Forte and Bruckman 2006). In addition, writing wikibooks helps learners not only to generate collective knowledge but also to review a chronological history of how knowledge evolves in academic topics (Mindel and Verma 2006). This affordance creates sophisticated performances of understanding (Wiske et al. 2005) and eventually fosters critical thinking through collaborative reflection (O’Shea et al. 2007; Reich et al. 2012).

Hence, researchers have investigated the potential for learning in wikis in authentic instructional situations using various types of data and data analysis methods. Ertmer and her colleagues (2011) examined 346 pre-service teachers’ confidence and perceived value for participating in cross-cultural wiki collaborations with pre- and post-survey data triangulated with focus group interview data. O’Shea and his colleagues (2007) measured 260 students’ perception and participation levels in surveys regarding the usefulness between wikibooks and other traditional tools. Forte and Bruckman (2006) conducted surveys and interviews to investigate links between wiki publishing experiences and writing-to-learn with scaffoldings in 42 undergraduate students’ government course and also analyzed students’ quantitative participation observed in wikis.

For the promotion of critical thinking via wikis, Mandernach (2006) suggested practical scaffolding strategies that instructors can use in their classrooms as follows: (a) brainstorming, (b) planning of learning activities, (c) document editing, (d) perpetually updated lists, (e) bulletin boards, (f) collaborative experiments, (g) informational debates, (h) teaching network literacy, and (i) ongoing revisions, changes, and modifications. He argued that wikis could allow students to overcome the barrier between content creator and content consumer through authentic interactivity and collaboration. Snodgrass (2011) also suggested a blended learning method combining wikis with in-class activities as a scaffolding strategy to enhance student collaboration and development of critical thinking skills. Snodgrass conducted research on the implementation and evaluation of wiki activities to promote critical thinking and clinical reasoning for undergraduate physiotherapy students. In blended learning, the instructor created a private wiki where students could share their patient cases in groups and communicate with their clinical mentors who were practicing clinicians relevant to each patient case. She argued that wikis could facilitate collaboration among students and enhance their learning of complex critical thinking skills, ensuring demonstrated reasoning skills in class.

Nevertheless, most wiki studies in education have been conducted as a single case study with limited data sources and analysis methods such as measuring students’ perceptions in a survey. Although these studies focused only on selected dimensions of wiki creation, the lack of multiple data sources and analysis methods can make it difficult to explicitly address the quality and quantity of actual learning experiences and outcomes in dynamic wiki collaborations. More research is needed to directly contribute to our knowledge of how to design effective instructional strategies to promote critical thinking in wiki-based learning environments.

Therefore, this study investigated how to scaffold students’ critical thinking skills in wikibook creation as a learning task and analyzed the quantitative and qualitative data comparing both perceived and observed levels of critical thinking and participation between two cases. Findings would be beneficial to understand how different degrees of scaffolding influence the development of critical thinking and participation within wiki collaborations as well as how computer-mediated discourse analysis (CMDA) can be used to compare wikibook modules between two cases. Two research questions were addressed:

  1. 1.

    How did students participate in wikibook creation with enhanced and minimal scaffolds?

  2. 2.

    What level of critical thinking did students display in wikibook creation with enhanced and minimal scaffolds?

Method

Research Design

This study used a multiple-case study research design (Lincoln and Guba 1985; Yin 2003) and a mixed methods research design (Teddlie and Tashakkori 2009) with CMDA including descriptive statistical analysis and content analysis (Herring 2004). As a theoretical framework of the research approach, an exploratory multiple-case study research design (Lincoln and Guba 1985) was adopted after carefully considering Yin’s (2003) three conditions of case study research: (a) contemporary phenomenon within its real-life context, (b) multiple sources of evidence, and (c) unclear boundaries between the phenomenon and the context. A bounded system can be a program, event, or activity (Cresswell 1998). This study’s boundary system was defined as a wikibook project with scaffoldings for the development of students’ critical thinking skills in wiki collaborations. According to Yin (2003), a multiple-case study approach can be selected to “predict contrasting results but for predictable reasons, i.e., a theoretical replication” (p. 47); which is appropriate for this study to investigate two wikibook creation projects with different levels of scaffoldings: enhanced scaffolding case (ESC) and minimal scaffolding case (MSC).

Both quantitative data (i.e., wikibook module histories, surveys) and qualitative data (i.e., wikibook modules, documents, interviews) in a mixed methods research design were used to analyze a variety of learners’ online participation and development of critical thinking as well as to confirm the findings in two cases from different perspectives. Particularly, CMDA was employed to analyze observed critical thinking levels and online participation patterns in wikibook modules. The core of CMDA is the log analysis of verbal interactions (e.g., characters, words, utterances, messages, archives) (Herring 2004). According to Herring (1996, 2004), it is important in the methodology of CMDA to analyze online discourse patterns using both quantitative means (e.g., coded and counted discourse phenomena, summarized frequencies) and qualitative means (e.g., illustrated and discussed observation of discourse phenomena) to identify and confirm patterns in message structure. Regarding both structural and semantic phenomena of discourse behaviors in wikibooks, two levels—participation level and functional moves level—were selected among Herring’s five levels of CMDA: participation, structure, functional moves, interaction, and social behavior domains. Participation level analysis described online behavior patterns of participants in the process of wikibook development. The functional moves level analysis revealed the quality of critical thinking skills of participants as a type of content analysis.

Cases and participants

One wikibook project in each of two face-to-face graduate courses was selected at a large state university in the Midwestern United States: (a) Enhanced Scaffolding Case (ESC) and (b) Minimal Scaffolding Case (MSC). ESC provided students with enhanced scaffolds to support their critical thinking development in wikibook creation while MSC presented only minimal scaffolds. ESC included 13 graduate students (10 master and 3 doctoral students) in an educational psychology course while MSC had 8 graduate students (5 master and 3 doctoral students) in an educational technology course.

Scaffolding

Participants were gradually exposed to increasingly difficult steps of wikibook development through the entire semester. One instructor and a graduate assistant provided ESC and MSC participants with enhanced and minimal levels of scaffolding each (see Table 2).

Table 2 A comparison of scaffoldings between ESC and MSC
Full size table

Job aids and critique guidelines

In the beginning of the semester, ESC participants received both job aids for creating wikibooks and critique guidelines of the class activities that the instructor and graduate assistant specifically designed for their wikibook project based on Collison et al.’s (2000) scaffolding strategies for critical thinking. MSC participants received only job aids.

Critiques of an existing wikibook

As a first task, ESC critiqued an existing wikibook on emerging perspectives on learning, teaching, and technology, which a large Southern state university’s students developed as part of their class project. Individual students critiqued each chapter based on the given critique guidelines. Their critiques, in Word document format, were given to their matched critical friends in the same class as well as to their instructor. After revision, each set of students’ critiques was posted on Wikispaces (http://www.wikispaces.com) by the graduate assistant; this site is similar to their wikibook website used for the final project (http://www.wikibooks.org). The MSC project did not have any critique activities with the existing wikibook.

Editing an existing wikibook

The second task in ESC was to edit an existing wikibook on learning theories and theorists by using actual wikibooks’ tools that students would use for their final task. This wikibook was originally created by graduate students at a Midwestern university as a class activity. ESC students practiced how to edit the wikibook in a computer lab during class with the instructor’s demonstration. MSC participants did not have a chance to edit any existing wikibooks.

Writing a wikibook

As the final task, ESC students created their own wikibook on learning theories. Students selected their own chapter topics and wrote the contents individually in Word document format. After finishing their first chapter drafts, they provided assigned critical friends with individual feedback via email. The instructor gave students his individual feedback via email as well. Once revised, students posted their final chapters to their official class wikibook website. They edited each other’s chapters in a computer lab during class and were also encouraged to continue editing after class on a voluntary basis.

MSC participants developed their own class wikibook on the topic of Web 2.0 learning technologies. Before selecting chapter topics, they brainstormed in class and together created a master list of chapters on emerging learning technologies. Once a master list was finalized, they selected their own chapter and title individually. Two of the eight students in MSC voluntarily chose to author two wikibook chapters each. Individuals wrote their chapter proposals and the instructor reviewed them. At the end of the semester, the students posted their final chapters to the official class wikibook website. They also had a computer lab session to edit peers’ wikibook chapters and were encouraged to offer feedback to each other after class.

Data sources and collection

To ensure valid and reliable conclusions of the study, four data sources were employed: (a) wikibook modules, (b) surveys, (c) documents, and (d) interviews (See Table 3).

Table 3 Overview of data sources, data collections, and data analysis methods
Full size table

Wikibook modules were used as the primary data source to investigate the patterns and levels of students’ online participation and critical thinking in quantitative and qualitative data analysis methods of CMDA. Thirteen ESC students created 13 chapters and eight MSC students developed 10 chapters. Surveys were conducted at the end of the semester to analyze students’ perceived levels of participation and critical thinking in wikibook development. All of the 21 participants in both cases completed an online survey. The survey consisted of 24 closed items in three sections (See Appendix 1): (a) twelve multiple-choice items on general demographics and prior experience with wikis, (b) six 5-point Likert items (1 = strongly disagree to 5 = strongly agree) based on Driver’s (2002) and Wise’s (2007) surveys to ask students about their perceived participation as authors and peer editors, and (c) six 5-point Likert items based on Collison et al. (2000) research to measure critical thinking. Cronbach’s alpha coefficients in this study were 0.81 for three author participation items, 0.74 for three peer participation items, and 0.90 for six critical thinking items. Documents, i.e., reflection papers, were collected to triangulate the findings from the survey and interviews. Twenty-one reflection papers were collected: 13 papers in ESC and 8 in MSC. After clarifying themes and patterns based on initial data analyses, follow-up interviews in an open-ended and semi-structured format were conducted with member checking (Stake 1995; See Appendix 2). Five face-to-face interviews were conducted with volunteers: three interviews in ESC and two in MSC.

Data analysis

Wikibook module data analysis: two levels of CMDA

To identify and confirm patterns in message structure of wikibook modules, two levels of CMDA (Herring 1996, 2004) were conducted: (a) participation level analysis describing participants’ online behavior patterns and (b) functional moves level analysis revealing their critical thinking levels observed in wikibook development. To reduce the subjectivity of coding, two coders participated in data analysis for both levels. During the initial content analysis, they clarified and modified the original coding scheme together and identified logical chunks when necessary based on emergent patterns in the data. Once coding schemes were developed, coders analyzed data independently and then compared results. Disagreements about assigned codes were discussed and resolved until both coders reached 100 % agreement, considering inter-rater reliability in content analysis schemes (DeWever et al. 2006).

The total number of words and the average number of words added and deleted per wikibook chapter and per participant were calculated first to examine the general quantity of online participation in both cases (Herring 1996). To investigate in-depth participation patterns, two coders conducted the initial content analysis with all of the ESC wikibook modules until they discovered the common indicators of emergent behavior patterns. The coding schemes were not predetermined before the analysis. Instead, the schemes were continually refined though the coding process of the initial analysis of wikibook modules. Example units for each coding scheme were identified and used to increase the consistency of the classification. While further defining and confirming each category, the initial examples were used to guide the analysis. Through the constant-comparison method (Lincoln and Guba 1985) in initial content analysis, two wikibook activities were categorized: (a) writing and editing participants’ own wikibook chapters as authors and (b) editing peers’ wikibook chapters as contributors. In terms of externalization and internalization in knowledge-building (Nonaka and Konno1998; Wise 2007), editing peers’ chapters as contributors can be the process of externalization through collective reflection while writing and editing authors’ own chapters is the process of internalization through conscientious practice. Writing an individual wikibook chapter as author was a required and graded task while editing peers’ wikibook chapters was an optional task in both cases.

Identifying the appropriate unit of analysis is fundamental in CMDA (Herring 2004). Three types of units that participants commonly added or deleted in wikibook modules were clarified through the initial content analysis: (a) Utterance units, (b) Object units, and (c) Layout units (see Appendix 3). An utterance unit is a logical and thematic chunk of text treated as a macrosegment (Herring 1996), which is a paragraph in this study. A paragraph included a wide range of body texts in wikibook modules, from a single sentence to many. Generally, an utterance unit was separated from the previous or the following paragraph by a break line, which authors inserted between the different logical and thematic chunks of text. An object unit is any object other than a paragraph such as titles, references, URLs, images, authors’ names, and tables. A layout unit is a spacing or indentation among utterance and object units such as changing spacing between two paragraphs. Unlike the other units, the frequency of layout units was counted per module due to its unique features. All three types of units (utterance, object, and layout) were counted for participation level analysis in both cases. However, in functional moves analysis, only utterance units directly related to critical thinking levels in wikibook development were considered.

Writing a wikibook establishes a cycle of inputting new content and editing existing content. For investigating the actual participation levels in wikibook development, it was essential to compare the frequencies and ratios of new to old content units. Therefore, a coding scheme of new and old content units in six degrees was designed based on the ratio of words added as new content in each utterance or object unit: (a) New content units, (b) Mostly new content units, (c) Half new and half old content units, (d) Mostly old content units, (e) Old content units, and (f) Deleted content units (see Appendix 3). Layout units were not included when assessing new and old content because layout units did not contain any content. Also, six components of object units were identified as another coding scheme in participation level analysis: (a) Titles including subtitles; (b) References; (c) URLs; (d) Images; (e) Authors’ information including names, titles, and affiliations; and (f) Tables.

To investigate the quality of critical thinking skills displayed in wikibooks development, functional moves level analysis was conducted only with utterance units (Herring 1996), i.e., paragraphs in the body text of wikibook modules. The boundaries of an utterance unit ranged from a completed or uncompleted sentence to completed or uncompleted paragraph. For a coding scheme in functional moves level analysis, Greenlaw and DeLoach’s (2003) critical thinking levels were adapted (see Table 4). Their indicators and distinctive characteristics of critical thinking are theoretically integrated into the meaning of the critical thinking construct, as shown in the multiple literature reviews (Brookfield 1987; Ennis 1985; Paul 1990).

Table 4 Critical thinking levels adapted from Greenlaw and DeLoach (2003)
Full size table

Following emergent patterns in the initial data analyses and coder trainings, the modifications of the coding scheme were revised repeatedly. Two experts reviewed the final coding scheme. Regarding the contradictory categorization cues in functional moves level analysis, Garrison et al.’s (2001) two heuristics were adopted: “(a) Code Down (i.e., earlier category) if it is not clear which phase is reflected and (b) Code Up (i.e., later category) if clear evidence of multiple phases are present” (p. 5). As a result of each case, descriptive statistics were provided to illustrate and compare the levels of critical thinking.

Survey, interview, and document data analysis

To refine the survey instrument, factor analysis was conducted with a total of 55 participants’ survey data: 21 in ESC and MSC and 34 in other wikibook projects. Two of three survey items in the high correlation were removed from the initial survey items adapted from Collison et al.’s study (2000) to measure critical thinking. The final survey included 24 items in total (See Appendix 1). As explained in the section of data sources and collection, Cronbach’s alpha coefficients were above .70, which were acceptable according to Nunnally (1978). Reflection papers and interviews were analyzed to discover reasons for online participation and critical thinking patterns. Main code categories and sub-code categories were culled from interview transcripts and reflection papers following Lincoln and Guba’s guidelines (1985).

Validity and reliability

To increase the validity and reliability of this case study, Ying’s five tactics (2003) were employed: (a) methodological triangulation, (b) data triangulation, (c) investigator triangulation, (d) internal consistency of the questionnaire, and (e) member checking. Methodological triangulation uses multiple methods to investigate a single problem (Denzin and Lincoln 2000). As a multiple-case study, a constant comparative method (Lincoln and Guba 1985) and CMDA (Herring, 2004) with descriptive statistical and content analysis were adopted. For data triangulation in a case study, Ying (2003) suggests six major sources of evidence: documentation, archival records, interviews, direct observation, participant-observation, and physical artifacts. Also, Denzin and Lincoln (2000) recommend a variety of sources of evidence from three subtypes including time, space, and person. This study collected and analyzed multiple data sources from both qualitative and quantitative data sets: histories of wikibook modules (time-triangulation) and data of surveys, interviews, and documents (person-triangulation) from the two courses (space-triangulation). Investigator triangulation involves the use of several different researchers or evaluators (Denzin and Lincoln 2000). This tactic can increase the interpretative base of the research and show the elements of phenomenon, which may be hidden to a single researcher. The author collaborated with two experts in wiki-based technology research who provided suggestions from research design to data analysis. Also, two coders independently participated in content analysis. Regarding the internal consistency reliability of the questionnaire, Cronbach’s alpha was employed. Although this study’s survey items were developed from existing questionnaires with high Cronbach’s alpha coefficients, reliability coefficients were calculated again after performing factor analysis to increase the validity of the questionnaire. To validate interview data, member checking was adopted. This allowed interviewees to provide their own critical observation and interpretations, which may be overlooked by the researcher (Stake 1995). Transcripts were provided to interviewees to confirm the intentions of their actions and words.

Results

Students’ participation in wikibook creation

Due to the effects of enhanced scaffoldings, ESC students were initially expected to show higher levels of participation in both authors’ and peers’ wikibook chapters than MSC students. However, some differences were found between students’ perceived and observed participation levels in both cases.

Perceived participation levels in surveys, documents, and interviews

According to the survey results, ESC students perceived that they participated more actively in the wikibook project and engaged more in the topic and content than MSC students did (see Table 5). For example, ESC students reported that they put forth relatively more effort to create their own chapters as authors (M = 4.7, SD = 0.6), compared to MSC students (M = 3.3, SD = 1.4). ESC students evaluated that their own and peers’ engagement levels were equally active (M = 4.2, SD = 0.8 and M = 4.2, SD = 0.7, respectively) while MSC students assessed that peers engaged a little more actively (M = 3.8, SD = 0.9) than themselves (M = 3.3, SD = 0.4).

Table 5 Perceived participation levels in surveys
Full size table

ESC reflection papers did not include any comments on their participation, such as time and effort in editing, as authors or peers. However, half of MSC reflection papers indicated that they did not participate in peer editing as actively as they could have.

During the interviews, all three ESC students rated their overall participation levels as very active. However, neither MSC interviewees described themselves as active participants though they engaged in the topic and content relevant to their interests. In terms of peer editing in MSC, one interviewee mentioned: “I don’t think there was a lot of feedback… People didn’t have time or didn’t follow up with each other” (Participant A). Interestingly, all interviewees in both cases answered that the techniques they used to edit peers’ chapters were different than those they used as authors. In their own chapters, they used Microsoft Word to write initial drafts and check out grammar and spelling errors though they directly edited peers’ chapters from the wikibook website without using any word processors.

Observed participation levels in wikibook modules

Content analyses of wikibook modules also revealed that in both cases, students participated more actively as authors. Overall, they added and deleted more words in their own chapters, compared to those in peers’ chapters (see Table 6). They added similar amounts of words per author’s chapter (2,943 words in ESC and 2,798 words in MSC).

Table 6 Added and deleted words in wikibook modules
Full size table

However, MSC added 288.2 words per peer’s chapter while ESC added only 82.8 words. Furthermore, MSC deleted extensively more words in both authors’ and peers’ chapters. MSC deleted 507.3 words and ESC deleted only 2.8 words per authors’ chapter. MSC deleted 136.2 words per peer’s chapter but ESC deleted 16.8 words.

Both cases tended to change more titles and references in object units of wikibook chapters rather than body texts, i.e., utterance units, except MSC peers’ chapters (see Table 7). MSC edited more utterance units (63.5 %) than object units (33.1 %) and layout units (3.4 %) in peers’ chapters.

Table 7 Frequencies and ratios of types of units in wikibook modules
Full size table

Regarding the utterance and object units, mostly new content units dominated in authors’ chapters (69.6 % in ESC and 61.2 % in MSC) as shown on Table 8 and Fig. 1. In peers’ chapters, however, old content units (57.1 % in ESC and 40.6 % in MSC) and mostly old content units (28.6 % in ESC and 47 % in MSC) were majorities (see Fig. 2).

Table 8 Frequencies of new and old contents in utterance and object units
Full size table
Fig. 1
figure 1

Ratios of new and old content units in authors’ wikibook chapters

Full size image
Fig. 2
figure 2

Ratios of new and old content units in peers’ wikibook chapters

Full size image

Students’ critical thinking development in wikibook creation

The ESC was anticipated to show higher levels of perceived and observed critical thinking in authors’ and peers’ wikibook chapters, and indeed ESC displayed relatively higher levels of critical thinking as authors and peers than the MSC did. However, two similar patterns of participation were found in both cases: (a) higher levels of critical thinking in authors’ chapters and (b) lower levels of critical thinking in peers’ chapters. Students tried to write more coherent statements with supporting evidence as authors, but they tended to make minor changes such as grammar and spelling corrections as peers.

Perceived critical thinking levels in surveys, documents, and interviews

In both cases, participants evaluated in surveys that creating a wikibook helped them develop critical thinking skills overall (see Table 9): M = 4.5, SD = 0.7 in ESC and M = 4.1, SD = 0.6 in MSC. ESC valued more about peer feedback than MSC did in terms of identifying directions of writing (M = 4.1, SD = 1.0 in ESC and M = 3.1, SD = 1.0 in MSC) and considering the topic from various perspectives in wikibook creation (M = 4.2, SD = 1.1 in ESC and (M = 3.0, SD = 1.1).

Table 9 Perceived critical thinking levels in surveys
Full size table

Reflection papers also reported that wikibook projects improved student critical thinking skills in many different ways. For example, six ESC participants mentioned that they could review their learning processes by creating wikibook chapters: “I found authoring a wikibook invaluable to my learning experience in the way that it helps me reflect on what I have learned and how I will integrate it into my teaching” (Participant S).

Seven ESC students pointed out that peer feedback gave them the opportunity to think critically about their own writing as well. One student stated that:

The peer feedback… allowed me to take a step back and not only critically consider someone else’s work, but realize how I could improve my own—keeping me engaged and motivated. The process as a whole was very interesting and only reinforced my understanding that learning may be best understood as reciprocal (Participant K).

Similarly, five ESC participants discovered that they could share their ideas and interests not only by writing their own chapters but also by reading their peers’ chapters. This knowledge-sharing experience helped them evaluate and reflect upon their own learning approaches within the educational community. One student explained: “This class has inspired me to continue to do well, to build a stronger foundation of what it takes to ‘learn,’ and to become a productive and inspiring member of the education community at large” (Participant H).

Furthermore, most ESC participants attempted to connect their learning experiences from the wikibook project with their future teaching based on the principles of learning theories. This belongs to the third phase of Garrison’s critical thinking, i.e., Integration (2000).

On the other hand, MSC students expressed their general positive learning experiences in wikibooks related to academic writing and thinking: “I have never seen myself as a good writer of academic papers, let alone for publication, but Wikibooks seemed a good start to develop confidence and, of course, a product” (Participant M). “After seeing the wikibook chapters of other students, I see that I was probably sticking too much to the academic realm. I see now that it is appropriate to break out of my rigid academician thinking” (Participant M2).

In terms of peer editing, however, all interviewees in ESC and MSC reported that they attempted to think and write critically only for their own chapters, but not for peers’ chapters. As authors, they tried to support statements with solid evidence required in academic writing; as peers, they verified only grammar and spelling. They initially used Microsoft Word for their chapters, but edited peers’ chapters directly on their wikibook websites.

Observed critical thinking levels in wikibook modules

Both cases displayed relatively higher levels of critical thinking in authors’ chapters (see Table 10). However, regarding the ratios of utterance units, ESC authors showed higher critical thinking levels than those in MSC (see Fig. 3). For instance, ESC participants tended to develop their authors’ chapters based on more theoretical inferences of Level 4 (41.0 %) and basic analyses of Level 3 (18.2 %) and simplistic alternatives of Level 2 (26.8 %). MSC authors showed relatively fewer theoretical inferences of Level 4 (27.6 %), more unilateral descriptions of Level 1 (19.9 %), and more non-critical thinking units of Level 0 (18.3 %), i.e., checking grammar and spelling.

Table 10 Frequencies of critical thinking levels in utterance units
Full size table
Fig. 3
figure 3

Ratios of critical thinking levels in authors’ wikibook chapters

Full size image

Overall, peer editing in both cases showed low levels of critical thinking (see Fig. 4). Nevertheless, regarding the ratios of utterance units, ESC participants displayed relatively higher levels of critical thinking in peers’ chapters (8.2 % in Level 4 and 18.0 % in Level 2) than MSC did (1.6 % in Level 4 and 8.6 % in Level 2). This is an interesting finding because MSC peers showed more active participation levels with more words added and deleted per peer’s chapter (see Table 6).

Fig. 4
figure 4

Ratios of critical thinking levels in peers’ wikibook chapters

Full size image

Discussion

Enhanced scaffoldings in wikibook creation: why did ESC show higher critical thinking levels but lower participation levels in the wikibook?

Multiple data showed that enhanced scaffoldings helped students increase their confidence through the wikibook project, think critically in writing chapters, and avoid trial and error with a new online tool. For example, five ESC reflection papers stated that students’ confidence was increased through step-by-step activities and peer feedback during the project. They explained that critique guidelines helped them “to get the picture of how to do and what to do” (Participant K). Also, two ESC interviewees reported that the instructor’s detailed guidelines and feedback reduced their initial tension and stress stemming from the challenge of new wiki activities.

Furthermore, survey results supported that enhanced scaffoldings were more helpful to think critically (M = 4.3, SD = 0.6 in ESC and M = 3.8, SD 0.9 in MSC; see Table 9). For instance, exchanging feedback with critical friends helped ESC students reflect on the issue from different perspectives:

My understanding of critique was just to point out the weakness. Later, my partner told me that you did not only need to refer [to] their shortcoming, but also you needed to find some strengths… from critique of other’s work, I thought about what I was doing on my own project and what kind of point I needed to avoid (Participant L).

One reflection paper also clarified that: “Guidelines helped my thinking… I need to consider very carefully about the structure, the contents, and also the references because the reference is something you need to refer after you read that and maybe you can do some further reading based on the references” (Participant A).

In addition, ESC students reduced time and effort in creating drafts of chapters and avoided technical difficulties in wikibooks. This positive effect of enhanced scaffoldings eventually decreased student participation in quantitative measurements. For instance, MSC deleted extensively more words in its wikibook: 1,810 % more in authors’ chapters and 811 % more in peers’ chapters compared to those in ESC (see Table 6). MSC changed more body texts, i.e., utterance units, while ESC edited more titles and references in object units and changed more webpage formats, which are layout units. Also, MSC showed more mostly old content units in terms of the ratios of the units. Meanwhile, ESC tended to make minor changes, mostly to titles and references after copying and pasting content texts into authors’ chapters. This is similar to proofreading in academic writing. An explanation for this might be that enhanced scaffoldings exposed learners to writing difficulties and technology issues with a new tool in the beginning stage and helped them reduce time and effort for their final task eventually (Rosenshine and Meister 1992; Vygotsky 1978). As a result, enhanced scaffoldings supported effective and efficient learner performances and decreased their frustration levels in wikibooks.

The purpose of the study was to investigate how to scaffold the development of critical thinking skills in wiki collaborations. In order to measure students’ critical thinking levels displayed in wikibooks using CDMA methods (i.e., functional moves levels), their online behavior patterns observed in wikibook modules (i.e., participation levels) had to be analyzed first for identifying the appropriate unit of analysis (i.e., an utterance unit) (Herring 1996, 2004). ESC students were initially expected to show high critical thinking levels as well as high participation levels in wikibook creation due to the effects of enhanced scaffolding. However, CMDA results revealed the opposite findings, i.e., high critical thinking levels but low participation levels in ESC in terms of the ratios of utterance units. This study tried to explain possible reasons for the unexpected results using qualitative data analysis methods for documents and interviews, but did not statistically analyze the relationships between individual learners’ critical thinking and participation levels in wikibooks. It would be necessary to consider this quantitative data analysis method in follow-up studies.

Differences between authors’ and peers’ wikibook chapters: why did ESC and MSC show low participation and critical thinking levels as peers?

Students actively participated in authoring their own chapters whereas they passively participated in editing peers’ chapters in ESC and MSC. All interviewees in both cases explained that personal knowledge about and interest in the chapter topic lead to their active participation as authors. Also, they tried to improve the quality of writing in terms of academic writing, content accuracy, and solid references in their chapters. This is because all of them were aware that people around the world could read their chapters and recognize them as authors: “This [wikibook] is an official Internet website. So, everybody can see what my project is and that made me focus on grammar and the contents” (Participant K).

However, they did not participate actively as peers. They answered that peer editing was an optional activity and they were struggling with the lack of time at the end of the semester. All MSC interviewees felt uncomfortable with peer editing because it could hurt peers’ feelings. In addition, four MSC reflection papers reported that students hesitated in peer editing for emotional reasons: “I found myself feeling very uncomfortable and invasive editing my classmates’ work, which may be due to the fact that the authors and I know each other. To me, these authors ‘owned’ the ideas presented in their assigned entry space” (Participant L).

These reasons can be explained by Gray’s (2004) barriers of knowledge-sharing in online communities which eventually reduced the quantity as well as the quality of peer editing in wikibooks. Also, this hesitation aligns with Grant (2009)’s article, “I DON’T CARE DO UR OWN PAGE!” addressing how students’ individual ownership prevented wiki collaborations. Interestingly, however, one ESC interviewee reported that peer editing was like give-and-take, so he should have edited peers’ chapters because they already edited his.

Freedom or chaos with minimal scaffoldings in peer editing: why did one student with expertise in wikis and editing dominate MSC peer editing?

MSC participants’ critical thinking levels in peer editing were low, but their participation was very active in terms of the number of words edited in peers’ chapters; MSC added approximately 268 % more words and deleted 625 % more words in total even with fewer participants (see Table 5). Interestingly, wikibook content analyses discovered that only one female doctoral student performed over two-thirds of MSC peer editing. Her interview revealed that she had expertise in academic writing at universities as well as substantial knowledge and prior experience in Web 2.0 online communities such as Wikipedia and Second Life. Her motivation and expectations were exceptionally high even before starting the wikibook project. Advanced research has reported ‘relative unevenness in wiki contributors’ like her case (Ertmer et al. 2011). Ortega et al. (2008) discovered that only 10 % of editors contribute 90 % of Wikipedia contents. Carr et al. (2007) also argued that a relatively small number of contributors highly participated in a wiki. In this way, students’ strong motivation associated with their prior knowledge, interest in topic, and understanding about the culture in online knowledge-sharing communities can be an essential factor to explain their large contributions in wikis. Particularly, in a free learning situation without any required tasks, such as peer editing in MSC, this type of motivation would be more relevant to learners’ voluntary participation which is participatory learning in an informal learning situation (Bonk 2009).

Conversely, however, minimal scaffoldings were not supportive for the students who lacked prior knowledge about and experience with wiki-based tools. For instance, four MSC participants described in their reflection papers that they did not know what to do or how to begin their wikibook project, which eventually hindered their participation. They struggled with the lack of procedural, conceptual, strategic, and meta-cognitive scaffolds mentioned by Hannafin et al. (1999). This is often reported as one of major difficulties that beginners have faced in online knowledge-sharing communities (Gray 2004).

On the other hand, ESC participants made ‘relatively even contributions’ in the wikibook without any exceptional contributor in peer editing. This can be explained by the effects of enhanced scaffolding. All ESC students exchanged peer feedback on documents with critical friends several times before completing their final chapters; this could decrease their needs for providing additional feedback, such as editing in peers’ wikibook chapters. Multiple required tasks given as enhanced scaffoldings also might block degrees of freedom (Wood et al. 1976) that could allow individuals to extend the boundaries of exploration and discovery as voluntary contributors.

In addition, enhanced scaffoldings to improve critical thinking skills seemed to be more beneficial for learners who did not have prior experience in editing. For example, two doctoral ESC interviewees who were former English teachers with sufficient editing experience displayed low critical thinking but high participation levels as peers (e.g., checking grammar errors); both of them had very high critical thinking levels as authors. Interestingly, content analyses revealed that another interviewee who was a first year master student without any prior editing experience showed the highest levels of critical thinking among the ESC peers. During the interview, he reported that critique guidelines and activities presented as part of enhanced scaffoldings were very useful to understanding how to provide critical feedback based on theoretical evidence in peers’ chapters. Therefore, more structured guidelines and supports should be provided to new learners who do not have prior experience and expertise in wikibooks, not only for enhancing their participation by co-writing, but also for promoting their higher-order thinking by co-reflecting.

Limitations of the study

Four limitations of this study need to be considered. First, due to the nature of case studies with small sample sizes, the findings might not be generalized to the entire population. Second, the two cases had different course topics: ESC’s educational psychology and MSC’s educational technology. In both cases, individual students voluntarily selected their own wikibook chapter topics. All of them wrote the chapters as their final academic papers supported by solid references which are required in any graduate level course. However, ESC might have provided more theoretical inferences because of the nature of the course topic. Third, both cases were face-to-face classes with online tasks, i.e., blended learning. Direct face-to-face peer feedback in the classroom could not be controlled. Lastly, the researcher was involved in both wikibook projects to provide enhanced and minimal scaffolds. According to Stake (1995), this direct observation offers a researcher a greater understanding of the case; however, a researcher’s presence with participants could affect their online behaviors.

Conclusion

Wikibooks allow numerous individuals to generate content together as a knowledge-building community in informal learning. This study was designed to find instructional strategies promoting students’ critical thinking through co-writing and co-reflecting in wikibooks as a formal learning task. Results showed that students considered wikibooks as a useful online collaboration tool, but they did not actively participate in peer editing. Enhanced scaffolding including structured guidelines and critical feedback was helpful in enhancing critical thinking development within wikibooks, while minimal scaffolding focusing on technical guidelines increased online participation, particularly in peer editing. Interestingly, the MSC displayed similar participation patterns to those of wiki communities in general; only a few individuals with expertise made huge contributions to wikis. Interviews revealed that many MSC participants did not figure out how to start and what to do in their projects. In contrast, enhanced scaffolding was more effective for those who did not have prior experience with wikis or editing.

These challenges of realizing the potential of wikibooks in formal learning settings should be considered carefully. There are several suggestions about instructional strategies to enhance critical thinking in wikibook learning tasks. First, instructors should not provide only wiki-based learning environments to students without appropriate cognitive and technical scaffolds for their online collaborations. ESC’s step-by-step enhanced scaffolds can be presented: (a) exploring wikibooks with technical supports and job aids, (b) exchanging critiques of an existing wikibook with peers, (c) editing an existing wikibook, and (d) creating a wikibook. Promoting interaction among students in wikis is quite challenging, however; learners tend not to edit peers’ pages. According to Reich et al. (2012), only 11 of 63 educational wikis analyzed in their study showed any form of collaboration. Most students created content only for their own pages. For this reason, increasing individuals’ motivation in peer editing would be the first step for their active participation in wikibooks. Nevertheless, we should notice that the quantity of students’ peer editing does not guarantee the quality of their wiki collaborations in terms of critical thinking; students mainly corrected grammatical errors in peers’ wikibook chapters. It is hard to find the ideal balance between enhanced and minimal scaffoldings in wiki-based learning situations. How much structure would students need for completing their wikibook tasks successfully and promote their critical thinking skills in wiki collaborations? As shown in the results of this study, too much scaffolding might prevent their voluntary participation and contributions in peer editing. Too little structure could let them experience unnecessary trial and error to figure out how to write a wikibook rather than critical peer feedback. Instructors should carefully design appropriate levels of scaffolds in terms of encouraging students’ participation within co-writing as well as presenting critical feedback through co-reflection to maximize the benefits of wikibooks integrated into curriculum.

The platform of wikibooks has infinite possibilities as a scaffolding tool to promote critical thinking. Yet, there is limited research in this area, particularly regarding the content analysis to explain students’ critical thinking developed by co-creating wikibooks in formal learning contexts. This study provides preliminary results that students with enhanced scaffolds displayed relatively higher levels of critical thinking as authors and peers in the process of co-writing and co-reflection of wikibooks as a learning task.

In order to advance this study, several future avenues of research can be considered. First, team-based wikibook projects should be conducted. Writing a wikibook chapter was a required task for an individual but peer editing was optional in this study. Team activities, which create more active peer interactions in wikibooks, need to be designed as required tasks. Second, complex mechanisms of exchanging peer feedback should be analyzed at the multiple levels’ CMDA. One ESC interviewee reported that he gave feedback only to the peers who had already made contributions to his chapter. Wasko and Faraj (2000) indicated that this would be one of the barriers to knowledge-sharing in online communities. Through in-depth CMDA, we should investigate hidden barriers and enablers of peer editing within wikibook chapters. Last, relationships between scaffolds’ types and learners’ prior knowledge (Collins 1991; Scardamalia et al. 1984) should be examined with multiple quantitative and qualitative data. It is necessary to recognize students’ potential hesitations to participating in wiki collaborations, particularly when their prior experience and confidence are low.

Learners in the Web 2.0 era are no longer passive consumers of knowledge presented by instructors and educational systems (Kozma 2005). Instead, learners are now able to create their own content and share it widely in online knowledge-building communities such as wikis. This paradigm shift requires us to advance our pedagogical approaches, research agenda, and instructional strategies (Reigeluth 1999). Educational researchers and practitioners should enable learners to develop their own thinking and learning experiences through the use of Web 2.0 collaborations beyond the boundaries of traditional instruction.