Introduction

The nature of science education as advocated by the Next Generation Science Standards (NGSS) entails an integration of scientific and engineering practices, crosscutting concepts, and disciplinary core ideas (National Research Council 2012). Under these standards, K-12 students would “engage in scientific and engineering practices and apply crosscutting concepts to deepen their understanding of core disciplinary ideas” (National Research Council 2012, p. 10). By emphasizing integration rather than long lists of facts, the new standards attempt to avoid curricula that are “a mile wide and an inch deep” and to raise awareness of scientific practices aligned with real-world knowledge-creation processes, which comprise asking questions, developing models, analyzing data, obtaining information, constructing explanations, and so on (National Research Council 2012).

Grounded in decades of research on expertise (Bereiter and Scardamalia 1993), knowledge building as an educational approach is aligned with the current societal emphasis on knowledge production and innovation across various sectors (Scardamalia and Bereiter 2014). Guided by a set of 12 principles that emphasize real ideas and authentic problems, improvable ideas, collective responsibility for community knowledge, and epistemic agency (see Scardamalia 2002), knowledge building distinguishes itself by striving to transfer high-level cognitive responsibility to students. When applied in science education, it engages students in collaboratively building ideas to explain scientific phenomena (Bereiter 2012), a practice that is highly compatible with NGSS (National Research Council 2012, pp. 67–71). Studies of its application in science education have so far demonstrated its efficacy in facilitating conceptual change (van Aalst 2009; Chan et al. 1997) and student understanding of the nature of science (Caswell and Bielaczyc 2002; Chuy et al. 2010).

This article reports on a study aiming to facilitate science learning by engaging a knowledge-building classroom in promisingness judgments of ideas—a practice recognized as an important aspect of scientific inquiry and creative processes, but one that remains scantily explored in education. By introducing pedagogical and technological supports designed for promisingness judgments, the present study attempts to nurture students’ capabilities in this area so as to foster science learning.

Conceptual understanding and epistemic beliefs in science learning

Improving students’ grasp of conceptual ideas is a constant focus in science education. Literature highlights that grasping challenging scientific ideas often requires conceptual change, which occurs when a learner finds the existing conceptual framework inadequate (Posner et al. 1982). Indeed, science learning is not the acquisition of concepts, but a process of restructuring a coherent theory framework that connects concepts together (Carey et al. 1999), or a gradual shift from a learner’s initial naïve “mental models” to scientific ones (Vosniadou and Brewer 1992). Such a change may depend on shifts in the ontological category of a concept (e.g., from categorizing electric currents as matter to process) (Chi et al. 1994), or the rise of systematicity from loosely connected knowledge fragments (DiSessa 1988). These empirically acclaimed insights lead to instructional strategies that take students’ initial ideas seriously, focusing on eliciting cognitive conflicts that destruct learners’ preconceptions (Posner et al. 1982), or constructively using current conceptions to facilitate conceptual change (Linn 2008).

Related to efforts to promote conceptual change is the recognition of epistemic beliefs as an important mediator and also a product of science learning. Epistemic beliefs are generally defined as beliefs about the nature of knowledge and the process of knowing (Hofer and Pintrich 1997). Within a rich body of literature on this topic, multidimensional perspectives of epistemic beliefs, which have been widely accepted, recognize independent dimensions of epistemic development. For instance, one multidimensional framework recognizes source of knowledge, certainty of knowledge, development of knowledge, and justification of knowing as four dimensions of epistemic beliefs; one’s epistemic belief rests on a continuum in each of these dimensions (Conley et al. 2004).

Conceptual understanding and epistemic growth are interconnected (Duit and Treagust 2003; Cano 2005). For example, when encountering complex information, students who hold a naïve belief that “learning is quick” (e.g., “you can learn all information in a textbook during the first reading”) tend to apply superficial strategies for learning and thereby fail to integrate knowledge deeply (Schommer 1990). Compared to seasoned scientists, students are considered to hold inferior beliefs that are more absolutistic, objectivist, and nonconstructivist (Mason and Gava 2007; Stathopoulou and Vosniadou 2007). To promote science learning, therefore, much attention needs to be directed to scientific epistemic beliefs that mediate and accompany conceptual change.

The knowledge building approach

Knowledge building as an established constructivist pedagogy presents two novel approaches to science learning: (a) a theory-building approach to deep understanding, and (b) a community-oriented, discourse-driven approach to learning. The theory-building approach is informed by a strand of conceptual change research that embraces a “knowledge-as-theories” perspective. According to this perspective, science learning involves revisions of coherent structures grounded in persistent ontological and epistemological commitments (Özdemir and Clark 2007). Accordingly, effective teaching would encourage students to produce explanatory ideas and build them into more scientific ones (Carey and Smith 1993). Knowledge building, defined as continual improvement of ideas (Scardamalia and Bereiter 2003), embraces the theory-building approach and grants each theory or explanatory idea a trajectory of development. The focus on continual improvement encourages “design-mode thinking,” a mindset that stresses the knowledge-building potential of ideas rather than their current truth values (Bereiter and Scardamalia 2003). While many other science teaching approaches, such as argumentation (Kuhn 1993; Osborne 2010), are mainly concerned with arriving at facts or warranted beliefs, knowledge building treats “ideas as objects of creation, development, assembly into larger wholes, and application” so that they can grow (Scardamalia and Bereiter 2007, p. 14). Even though facts and the practice of argumentation are also critical, the proposition that ideas can be improved—instead of simply being rejected or accepted based on their truth values—affords an alternative means to engaging students in science learning. In line with the NGSS, knowledge building treats science learning as an endeavor to build scientific theories with increasing explanatory power and coherence; instead of emphasizing the acquisition of facts, knowledge building stresses building theories that account for accepted facts (Bereiter 2012). This enterprise of building increasingly powerful theories relies on design-mode thinking and constant efforts to look for promising directions pregnant with significant breakthroughs.

Besides the theory-building approach, knowledge building treats the advancement of knowledge as a community enterprise rather than the task of individuals. It attempts to rethink classrooms as knowledge-creating organizations in which the state of knowledge is determined by the community as a whole (Scardamalia and Bereiter 2014). Hence, knowledge building attempts to elicit students’ real ideas and treats them as epistemic artifacts that can be publicly shared and continuously improved by a community (Zhang et al. 2007). Knowledge-building discourse, supported by software environments such as Knowledge Forum (Scardamalia 2004), becomes a tool for supporting this community endeavor. To improve their ideas, students collectively take on high-level cognitive responsibilities (Scardamalia 2002)—for example, setting goals, monitoring progress, and making plans to address challenges—which are normally assumed by teachers in inquiry tasks and project-based learning. This practice of having students assume high-level cognitive responsibilities distinguishes knowledge building from other approaches such as project-based learning (see Chen and Hong 2016), and brings student work in closer alignment with authentic scientific inquiry, which is absent in simple inquiry tasks that are presently common in schools (Chinn and Malhotra 2002, p. 190).

Existent literature on knowledge building has documented its efficacy in promoting conceptual knowledge and epistemic beliefs in science learning (see Chen and Hong 2016, for a review). Researchers found that knowledge building mediates the effect of conceptual conflicts in conceptual change instruction (Chan et al. 1997), plausibly because knowledge building exposes students to the hypothetical nature of scientific theories and increases the chance of conceptual change with students (Vosniadou and Kollias 2003). Researchers also found that the student groups that present the strongest features of knowledge building—such as a sense of community, explanation-seeking inquiry, and efforts to interpret and evaluate information—achieved the most conceptual progress (van Aalst 2009; Lam and Chan 2008). As for epistemic development, knowledge-building classes have demonstrated progress in moving from viewing science as a simple, static entity towards believing scientific ideas are complex, tentative, and extendable (Caswell and Bielaczyc 2002; Lam and Chan 2008). In a study contrasting knowledge building with a project-based inquiry scenario, student in the knowledge-building condition were found to have deeper understanding in the nature of science (e.g., the nature of theoretical process, theory-fact differentiation, and the role of ideas in scientific progress) (Chuy et al. 2010).

Promisingness judgments in knowledge building

Promisingness is not an abstract idea, but an everyday term that can be applied in various scenarios. Promisingness simply means “the quality of being promising” (Oxford Dictionaries, n.d.). This definition applies in scenarios such as deciding which route is more promising for getting home quickly and which rookie player is most promising for a struggling sports team’s rebuild. So when the term promisingness is deemed difficult, it is more likely a result of its infrequent use rather than its semantic complexity.

The evaluation of promisingness is a natural component of creative processes. In creative problem-solving, identifying promising ideas is an important converging process following diverging idea generation (Treffinger 1995). In explaining the creative process, Gardner (1994) emphasizes the importance of promisingness in helping people attend to and then invest in “discrepant elements” in their work, before these elements manifest as fruitful in any way; in this case, promisingness plays a role in committing scientists to productive paths of scientific inquiry and guides their critical decision-making in circumstances where principled knowledge remains scant (Bereiter 2009). Experiencing successes and failures in making such judgments allows one to accumulate knowledge upon which promisingness judgments are based; having such experiences is thus a sound approach to improving one’s capability in making promisingness judgments (Bereiter 2002b).

Promisingness judgments are also an important element of knowledge building in schools (Bereiter 2002b; Bereiter and Scardamalia 1993), as knowing the limits of a community’s current understanding and finding promising directions for pushing those limits are important aspects of cognitive responsibility in knowledge building (Scardamalia 2002). As in real-world scientific laboratories, the need for identifying promising directions for inquiry is also vital for knowledge-building classrooms striving for continual idea improvement. Engaging students with this responsibility would help to strengthen the knowledge building pedagogy and bring it even closer to real-world science (Scardamalia and Bereiter 2003).

To enable students to make promisingness judgments, I and my colleagues initiated efforts to devise new technological tools and pedagogical supports and iteratively refine them following a design-based research approach (Collins et al. 2004). The larger design-based research initiative, of which this study was part, is focused on the twofold goals of (a) developing design frameworks to guide promisingness judgments interventions in classrooms (in the form of design solutions involving technological tools and pedagogical strategies), and (b) developing new domain theories of promisingness judgments (e.g., a theory of the challenges facing students who are making promisingness judgments, a model of promisingness judgments in the knowledge building context). It is characteristic of design-based research in general to have such twin goals (Edelson 2002). To achieve them, we first developed a Promising Ideas Tool as an add-on for Knowledge Forum, which is an online, community space for knowledge building (Scardamalia 2004). The tool was then piloted in elementary classrooms, to reveal the challenges faced by students with the concept of promisingness (Chen et al. 2011). In response to these identified challenges, we designed pedagogical principles and supports for knowledge-building classrooms, and conducted classroom interventions with designed pedagogical supports and a refined version of the tool (Chen et al. 2015). The first robust intervention demonstrated that students as young as 8 years old could grasp the essence of promisingness, apply it in their knowledge-building practice, and as a result, achieve greater knowledge advancement and closer collaboration (Chen et al. 2015). Further design-based research is needed to understand students’ engagement with promisingness and to explore the latent value of promisingness judgments for other aspects of knowledge building. In addition, reflecting the other side of the twin goals in design-based research, further work is also needed to improve the designed tool and principles in light of lessons learned from earlier studies.

The promisingness intervention

Building upon earlier iterations of the design-based research program on promisingness (Chen et al. 2011, 2015), an intervention comprising the following two parts was devised for the present study: (a) a refined version of the Promising Ideas Tool (PI), and (b) refined pedagogical supports to facilitate student understanding of promisingness and socio-cognitive processes of making promisingness judgments with PI.

Promising Ideas Tool

Knowledge Forum, a widely used knowledge-building environment, provides an online community space for continual idea improvement. In Knowledge Forum, students author notes, in which they contribute theories, evidence, opinions, and syntheses. Notes are organized under views, which provide rich visual and structural functionalities to explicate the relations among notes so that high-level inquiry structures become more explicit (see Scardamalia 2004). At the note level, students are provided with scaffolds as epistemic prompts; for example, a set of widely adopted theory-building scaffolds include “My theory,” “I need to understand,” “New information,” “This theory cannot explain,” “A better theory,” and “Putting our knowledge together.” At the view level, actions such as building on, referencing, and rising above are supported to facilitate idea connection, summarization, crisscrossing, and synthesis. Overall, Knowledge Forum provides a networked multimedia space for deepening inquiry within a community.

Grounded in the literature and prior design efforts (Chen et al. 2011, 2015), three aspects of promisingness judgments were supported by the new version of PI. Each aspect is described below, with special attention given to their supports for knowledge-building principles (italicized in the following text).

Idea tagging

Tagging promising ideas in Knowledge Forum represents an epistemic practice that foregrounds individual and collective reflection on community knowledge. This practice strengthens the principles of epistemic agency and community knowledge by bringing all community ideas under scrutiny for determining next steps. Using PI, a student can tag a text snippet in a note as being promising (see Fig. 1, left side). The tool provides a customizable set of “promisingness categories” to further mark tagged ideas (a new feature compared to the previous version). The default set of categories in the tool includes: “promising idea,” “unsolved problem,” “useful fact,” and “dead-end,” which roughly represent four states of promisingness. An optional text field is provided for the student to identify a content-based “promisingness criterion” for a tagged idea to further justify the choice of a promisingness category (another new feature in this version; see Fig. 1, right side). In a nutshell, the central purpose of idea tagging is to elevate potentially promising (and unpromising) ideas out of their local discourse contexts for further consideration.

Fig. 1
figure 1

Highlight an idea with the promisingness tool in Knowledge Forum. Left: clicking on the “Ideas” button in the note window will activate a set of highlighters for tagging ideas. Right: a pop-up window after a tagging action, prompting the student to provide a content-based criterion for this highlight

Full size image

Collective and individual reviewing

An idea aggregation window that lists all tagged ideas in a view provides an opportunity for review and deliberation by groups or individuals (see Fig. 2). Filtering and search functionalities—based on promisingness categories, criteria, or word match—were provided in this window to facilitate the review process (another new feature). Such a review process supports the principles of democratizing knowledge and symmetric knowledge advancement, by bringing each student’s highlights, which reflect distinctive interests and understanding, to focal discussion.

Fig. 2
figure 2

Idea aggregation window with category and criterion filters, a search box, and an Export button. Ideas are ranked by the number of “Hits” based on text overlap

Full size image

Idea exporting

Since the purpose of promisingness judgments is to define the next steps of the inquiry (Gardner 1994), an exporting feature has been introduced to PI (Chen et al. 2015). When reviewing the aggregated idea list, a student can choose several ideas to be exported to another workspace for further inquiry (see Fig. 3). Each time, selected ideas are exported to a single note as references, with pointers to the original notes (a new design in this version); the student can then edit this note to explain connections among exported ideas and meaningful next steps. In this way, PI facilitates sophisticated knowledge processes—such as rising above multiple ideas to create a synthesis, resolving conflicting explanations, and linking explanations with evidence—all making use of promisingness judgments made by students. A new set of epistemic scaffolds, including “We used to think,” “We found,” “Now we think,” and “Next we will,” were specifically designed to facilitate this exporting and forward-looking process. This exporting function enacts the principles of improvable ideas and rise-aboves by putting ideas into trajectories of continual improvement.

Fig. 3
figure 3

A synthesis note created based on identified promising ideas. Promising ideas exported from the idea aggregation list are shown as references (in italic at the bottom). Text surrounded by scaffolds (in yellow brackets) was written by students to synthesize these exported ideas and plan next steps (Color figure online)

Full size image

Pedagogical supports

To support the use of PI, pedagogical supports were first designed and described by Chen et al. (2015), and further refined in the present study, with three key elements described as follows.

Reinforcing design-mode thinking

Previous design experiments showed that without proper exploration of promisingness or design-mode thinking, students were likely to tag important-sounding facts as promising ideas—a clear reflection of a mode of thinking in traditional school practices that emphasize justification of ideas (i.e., arriving at true or justified beliefs) (Chen et al. 2011). However, to facilitate authentic scientific inquiry in classrooms, both design and justification modes of thinking are needed. On one hand, to justify or refute a scientific explanation students need to weigh arguments for and against it. On the other hand, students should also be given the opportunity to look for the “usefulness, adequacy, improvability, and developmental potential of [an idea]”—in other words, to venture into the design mode, which looks beyond the true–false dichotomy (Bereiter and Scardamalia 2003). An overemphasis on justification and justified beliefs is likely to throttle promising ideas, the fruitfulness of which might be manifested after long stretches of further development (Gardner 1994; Scardamalia and Bereiter 2007). Tolerance of error or imperfection at early stages might allow promising ideas—which are not necessarily true—to develop and assemble into significant ones. To help students venture into the practice of promisingness judgments, attention should be turned to design-mode thinking, which is normally overshadowed by the pursuit of justified beliefs in school practices.

Exploring the promisingness concept

Building upon previous studies (Chen et al. 2011, 2015), a class session was designed to engage students in exploring the promisingness concept, so that their prior understanding could be elicited and then advanced. During this session, students are not directly given a “definition” of promisingness. Rather, they are engaged in emergent discussion of pertinent questions such as “What does a promising idea mean?” and “How does a promising idea differ from an accepted fact?” Through emergent dialogues, students are encouraged to come up with their own questions, elaborate their understanding to peers, weigh different claims, reach consensus, and continue to refine their understanding of promisingness. The teacher’s role is to identify student conceptions that are more closely aligned with a conception of promisingness emphasizing “knowledge-building potential” and to encourage further exploration around it. Although this conceptual exploration could be introduced in the beginning of the intervention, it is expected that students continue to develop their understanding of promisingness throughout the intervention.

Making promisingness an essential constituent of knowledge building

After raising awareness about design mode and promisingness, the next goal is to configure the socio-cognitive setup to recognize promisingness as an important constituent of knowledge building. The essential goal here is to empower students to make choices about promising directions, in order to dictate knowledge-building discourse. To this end, a multiphase discourse design utilizing PI is proposed: (a) Review—students review knowledge-building discourse and identify promising ideas using the tagging function of PI, (b) Reflect—students collectively reflect upon identified promising ideas using the idea aggregation window, and (c) Refocus—students make choices about which ideas to refocus on and export them to new workspaces using the exporting function of PI. This Review-Reflect-Refocus (3R) wraps up the current phase of idea refinement and initiates a new one. The central goal, again, is to give promisingness judgments a prominent role in knowledge-building discourse, so that students’ epistemic agency and cognitive responsibility become further instantiated, and student choice becomes a determining factor for the community enterprise of knowledge building.

The present study

Building on earlier studies on promisingness, the present study tested the designed promisingness intervention. Going beyond prior work, this study introduced epistemic beliefs as important factors potentially related to promisingness judgments, in addition to classroom dynamics and scientific understanding explored earlier (Chen et al. 2015). The following research questions were addressed in the study: (a) To what extent could students’ understanding of promisingness be improved through the intervention? (b) How did students’ conceptual understanding change during the intervention? (c) How did students’ epistemic beliefs change during the intervention? and (d) To what extent did these changes correlate with each other?

Methods

Educational context and participants

This study was conducted in a K-12 school in Bogotá, Colombia, with students mostly coming from middle- or upper-middle-class families. This school was a bilingual school; all students could speak English fluently, and all of their science lessons were taught in English. This school has been involved in a Knowledge Building International Project (KBIP) for years. KBIP operates as a distributed network of knowledge-building classrooms, each of which enjoys much autonomy in terms of curricula and instructional design, but come together based on a shared appreciation of the knowledge building pedagogy. The focus of student work is on issues related to climate change, energy, and sustainable development (Laferriere et al. 2012).

Participants in the study were students from a sixth-grade class (n = 26) and their science teacher. No control class was included, because the present study, as part of a design-based research program, aimed for sustaining design innovations instead of confirming causations (Bereiter 2002a). Before this study, the teacher and students had several years of experience with the knowledge building pedagogy and technologies through the KBIP project. During the study, the sixth-grade class was studying a 10-week biology unit on “Population Growth,” as part of a broader group of topics related to biodiversity, including “Principles of Energy,” “Energy Transfer,” and “Environmental Problems.” There were four lessons each week, with each lesson lasting for 45 min. The knowledge building pedagogy was applied in the class, to the extent that students discussed their ideas about population growth and contributed notes to Knowledge Forum. The school culture emphasized knowledge-based learning goals in line with curriculum standards. In the study, the teacher chose the textbook, designed classroom activities in advance, and facilitated class discussions. Considerable time was spent ahead of the study to further align her teaching with the knowledge building pedagogy, in order to set the stage for students to make promisingness judgments.

Procedures

Two instantiations of promisingness judgments divided this study into three phases, each lasting for approximately 3 weeks (illustrated in Fig. 4). The decision to have students conduct two rounds of promisingness judgments was based on the length of the science unit, so that the class could produce enough new ideas to be examined in each promisingness judgments session. During Phase 1, the class started with typical idea refinement in knowledge building. Under the teacher’s guidance, the whole class discussed key concepts of population growth (e.g., food chains, carrying capacity of an ecosystem) and contributed explanatory ideas in Knowledge Forum. During the process, the researcher provided continual support for the teacher to reinforce design-mode thinking in the class (see the Pedagogical supports subsection for details). In the end of Phase 1, one 45-min class session was spent on exploring the promisingness concept, followed by another session focused on the collective process of Review-Reflect-Refocus (3R), during which students worked in pairs or triads to judge the promisingness of their community ideas using PI. Both sessions were co-led by the teacher and the researcher. In the final refocus effort, each student pair or triad exported a few promising ideas into a new note. Afterwards, the whole class started the second phase of idea refinement (see Fig. 4). Similar to Phase 1, another process of 3R happened in the end of Phase 2, giving rise to Phase 3 of idea refinement. In the final phase, no collective effort was made to conduct promisingness judgments; instead, students spent one session first reviewing their prior judgments individually and then discussing them as a whole group. Did ideas that had been identified earlier as promising turn out to be fruitful? Class discussion was focused on this question, and after the discussion, each student wrote a reflection note in Knowledge Forum. Because it was near the end of the semester, much time in Phase 3 had to be allocated to exam preparation.

Fig. 4
figure 4

Procedures of the present study. Two instances of PJ/3R (i.e., promisingness judgments instantiated by the Review–Reflect–Refocus design) divided the unit into three phases. In each phase, students participated in collaborative idea refinement. At the end of Phase 3, instead of conducting another round of PJ/3R, students reflected on successes and failures in their earlier judgments. Throughout the study, supports for design-mode thinking and conceptions of promisingness were provided as needed. Note: In this illustration, the time dimension is not properly scaled

Full size image

During the whole study, pedagogical supports to facilitate the conceptual exploration of promisingness were provided whenever needed, as were supports for knowledge building and design-mode thinking. Students were given a pretest at the beginning of this unit and an identical posttest at the end, each comprised a population growth knowledge test, an epistemic beliefs instrument, and a promisingness knowledge instrument, as described in the following section.

Instruments

Table 1 presents instruments, data sources, and analyses corresponding to the four research questions of this study. Instruments for assessing conceptual knowledge, epistemic beliefs, and understanding of promisingness included the following:

Table 1 Overview of research questions, instruments/data, and analyses
Full size table

Population growth knowledge test

To assess student understanding of population growth, a knowledge test comprised of three multiple-choice questions (worth one point each) and seven short answer questions (worth two points each) was developed based on curriculum standards, textbooks, and common misconceptions documented in literature (e.g., Barman et al. 1995). The knowledge test had a full score of 17.

Epistemic beliefs instrument

An established instrument developed by Conley et al. (2004) was chosen to measure the epistemic beliefs of students. This instrument contained twenty-three 5-point Likert-scale items (1 = strongly disagree; 5 = strongly agree), focusing on the following four epistemic dimensions:

  • Source (five items): beliefs about knowledge residing in external authorities (e.g., “Whatever the teacher says in science class is true”).

  • Certainty (six items): beliefs in right answers (e.g., “All questions in science have one right answer”).

  • Development (six items): beliefs about science as an evolving and changing subject (e.g., “Sometimes scientists change their minds about what is true in science”).

  • Justification (nine items): beliefs concerned with the role of experiments and how individuals justify knowledge (e.g., “Good answers are based on evidence from many different experiments”).

In the instrument, higher scores reflected more sophisticated beliefs, after adjusting reverse items.

Promisingness knowledge instrument

To probe students’ understanding of promisingness, an instrument containing three 5-point Likert-scale items (1 = strongly disagree; 5 = strongly agree) were constructed:

  • When I first come up with an idea to explain something, being correct is the most important thing.

  • I often try to come up with my own explanations different from those in textbooks when I learn science.

  • Scientists often make mistakes, and they’re good at learning from them.

These items were developed to estimate understanding of important aspects of promisingness. Item 1 targets students’ conception of knowledge-building potential (versus justified beliefs) conception. Item 2 probes the extent to which promisingness is tied to ideas found in authoritative sources for students. Item 3 targets students’ understanding of the source of promisingness knowledge.

Data analyses

In addition to student responses to the pre- and posttests, collected data also included video recordings of classroom dialogues and Knowledge Forum log files containing students’ online activities (e.g., notes written and ideas tagged by students) (see Table 1). The pre- and posttests were scored according to the answer keys. Analyses of Knowledge Forum activities and classroom dialogues are described below.

Knowledge Forum activities

Students’ Knowledge Forum contributions were collected and qualitatively analyzed. Content analysis was applied to notes and tagged promising ideas. An overview of this analysis is provided below:

  • Level of scientific sophistication. Knowledge Forum contributions containing student-generated explanations were first identified, including the following: proposing an explanation; supporting an existing explanation by providing a justification; improving an existing theory through elaborating, specifying details and using new evidence; and seeking a different explanation (Chuy et al. 2011). The quality of identified explanations was then evaluated based on a 4-point scientific sophistication scale from Zhang et al. (2007): 1—prescientific (containing a misconception while applying a naive conceptual framework), 2—hybrid (containing misconceptions that have incorporated scientific information), 3—basically scientific (containing ideas based on a scientific framework, but not precise) or 4—scientific (containing explanations that are consistent with scientific knowledge).

  • Quality of promisingness judgments. Agreement with adult experts was used as a proxy for the quality of students’ promisingness judgments. In the ratings of promisingness judgments, experts first read through the student discussions in Knowledge Forum so as to ground their ratings in the sixth-grade science context instead of scholarly research. Specifically, ideas highlighted under the category of “promising ideas” were analyzed in these two aspects: (a) level of promisingness—how promising is the idea within its knowledge-building context from a knowledgeable adult’s perspective; and (b) content-based promisingness criterion (i.e., the “promising for what” question)—the extent to which the criterion identified by the student relates to the tagged idea. These two aspects were rated on a 3-point scale (see explanations and examples in Table 2).

    Table 2 Coding scheme of promisingness judgments
    Full size table

Overall, two independent raters coded these Knowledge Forum notes and ideas and their agreement measured by joint agreement was 0.82.

Evolution of student conception of promisingness

In addition, videos of classroom discussions were transcribed and qualitatively coded (Burnard 1991), with a goal of tracking the change in students’ conception of promisingness. This analysis provided qualitative accounts of students’ understanding of promisingness and its evolution during the study. The videos also provided an opportunity to assess the enactment of designed pedagogical supports in this study.

Results

In this section, I start by giving an overview of knowledge-building activities and the enactment of the pedagogical design, followed by an analysis of the promisingness judgments that occurred in the study and the changes in student understanding of promisingness. I then present the development of students’ conceptual knowledge and epistemic beliefs, and finally I explore possible linkages between these changes and promisingness judgments. Since the present study followed the design-based research approach, results are focused on the potential usefulness of the design innovation rather than on establishing causation (Bereiter 2002a). Therefore, the results reported here explore changes (if any), instead of confirming differences, during the promisingness judgment intervention.

Knowledge building activities and enactment of the pedagogical design

In the process of using Knowledge Forum for online discussions in this study, students created three views, entitled “Welcome 6A,” “Inquiry Step 2,” and “Populations,” for the three phases respectively. An overview of notes and tagged ideas (under all four promisingness categories) is provided in Table 3. In Phase 1, the class started with a knowledge-building discussion in “Welcome 6A,” producing 89 notes. During the first instance of Review-Reflection-Refocus (3R) centering on promisingness judgments, student pairs and triads tagged 20 ideas in total; each student pair or triad created a synthesis note by exporting a selection of tagged ideas to the second view, “Inquiry Step 2.” In Phase 2, students wrote 64 notes and tagged 17 ideas. After another instance of 3R, students wrote another 51 notes in Phase 3 but did not gather for another 3R process.

Table 3 Overview of Knowledge Forum Activities
Full size table

The enactment of the designed pedagogical supports took place mostly in the first promisingness judgments session (see Fig. 4) that was focused on the exploration of the promisingness concept. Following the designed supports, students gathered to discuss key questions as a whole class (see the Pedagogical supports subsection above). Discussion of the first few questions uncovered students’ naive understanding of knowledge building and promisingness. First, when asked about what to do with questions posted in Knowledge Forum, student responses overwhelmingly focused on “answering” them, without mentioning more advanced knowledge processes such as analyzing, problematizing, and operationalizing a question. For example, one student said “if it’s too difficult, no one is going to answer it.” The overwhelming emphasis on providing answers exposed a passive, reactive question–answer dynamic in the class, in which high-level epistemic agency and responsibility remained detached from students (Dillon 1982).

Classroom discussion later moved onto discussing what is a “good” or “bad” idea, to ground the introduction of the promisingness concept. In line with their thinking on questioning, students’ initial responses focused on the “correctness” of answers. For example, one student said that “a good answer is precise and accurate.” At this point, student discussion became more vibrant, as competing ideas erupted in the same time: students explained ideas such as “sometimes one question can have multiple answers,” “judging whether one answer is good or bad depends on one’s personal opinion,” and “that needs trying and trying till the answer is complete.”

Recognizing that students became increasingly aware that tackling a question usually required long stretches of work, the researcher decided to introduce the concept of “promising ideas.” After making sure students understood the word “promising” and its Spanish equivalent, students worked in small groups to discuss this concept and recorded their thoughts on paper. They then gathered as a whole group and explored issues such as “whether promising ideas are all correct,” “what is a fact,” and “what we are going to do about identified promising ideas.” Their evolving understanding of promisingness is presented in the next section. Overall, regardless of challenges with students’ initial understanding, the first pedagogical session was productive in initiating students to think beyond truthfulness and recognize promisingness as an aspect of idea improvement.

Evolving understanding of promisingness and promisingness judgments

Knowledge of promisingness accumulates from experiencing successes and failures of promisingness judgments during creative processes (Bereiter 2002b). One important research question for the present study was whether students’ knowledge of promisingness improved during the intervention. Specifically, could students depart from a “truthfulness- or justification-oriented” understanding and move toward a “knowledge-building potential” account of promisingness?

Results of qualitative coding of classroom dialogues did reveal a gradual shift in students’ conception of promisingness. Four identified themes of understanding and their examples are presented in Table 4. In the early stage of the intervention, students’ intuitive understanding of promisingness overwhelmingly centered on truthfulness. As one student said, a promising ideas is “true,” or “You have the observation [to prove] that [it] is true.” As described in the previous subsection, this focus on truthfulness was later challenged by a relativist view, introduced by a few students who recognized that “[whether an idea is promising] depends on the person who writes the idea” or makes the judgment call. As the discussion went on, the relativist point of view led to the notion of possibility—that a promising idea is not necessarily true, but might ultimately lead to warranted ideas. Then student conceptions became further elaborated in later discussion. For instance, some students came to realize that promisingness judgments served to identify areas that were worth investing time on. “A promising idea is not the answer,” as one student noted, “but an idea leading to further discussion or investigation.” However, while some students were able to achieve such progress towards the conception of knowledge-building potential in one session, some others remained fixed upon truthfulness. Continual pedagogical supports were therefore provided, in a form similar to the first promisingness session. Overall, these findings indicated that through emergent conceptual exploration of promisingness, sixth graders could operate successfully on the basis of the knowledge-building-potential account of promisingness regardless of observable individual variations. A paired sample t-test assessing the change in the conception of promisingness in pre- and posttests agreed with the qualitative analysis. Results indicated an improvement from 3.48 (SD = 0.42) to 3.72 (SD = 0.48), t(24) = 2.03, p = 0.05.

Table 4 Facets of student conceptions of promisingness
Full size table

Students’ actual use of the Promising Ideas Tool provided an opportunity to examine their “in vivo” understanding of promisingness. To this end, quality ratings of students’ promisingness judgments were conducted by experts. Because no promisingness judgments were coordinated in Phase 3 (see Procedures), the first two phases were compared. Results of expert ratings on two aspects—idea promisingness and judgment of criterion—are presented in Table 5. The mean scores increased from Phase 1 to Phase 2: from the experts’ perspectives, students were moving from identifying ideas “worth exploring but unconducive to breakthroughs” (Level 2) towards ideas “leading to possible breakthroughs” (Level 3). However, t-tests did not confirm any significant improvement at the 0.05 significance level. Further inspection uncovered great variance among students, which triangulated with the qualitative analysis presented above. Therefore, analysis of students’ promisingness judgments indicated that even though their conception of promisingness had shifted over the course of the intervention, the changes that happened with their actual performance of the most important aspect of promisingness judgments was not statistically significant.

Table 5 Improvement of promisingness judgment performance
Full size table

Knowledge advancement

Domain knowledge test

A paired sample t-test was conducted to assess the change in student performance in the conceptual test. Results indicated a significant improvement, t(24) = 5.75, p < 0.001. The average score was improved from M = 5.44 (SD = 1.74) to M = 8.46 (SD = 3.16).

Level of scientific sophistication

An independent sample t-test was conducted to compare the scientific sophistication of explanations in Phase 1 and 2; the final phase was left out in this comparison because of its limited number of explanatory contributions. Results indicated that the scientific sophistication of explanations had improved significantly from Phase 1 to 2, t(44) = 2.02, p < 0.05. Notes moved from a hybrid level of scientific sophistication (M = 2.12, SD = 0.85) to a level closer to prescientific (M = 2.65, SD = 0.93).

Overall, analysis of both the knowledge test and conceptual contributions in Knowledge Forum indicated significant conceptual advancement during the study. No control class was available for comparison.

Changes with students’ epistemic beliefs

Confirmatory factor analysis (CFA) was first applied to test the consistency of the epistemic beliefs instrument from Conley et al. (2004). A test of the four-dimension hypothesis in CFA was sufficient (with a goodness-of-fit score = 0.94); four identified factors were properly loaded on related questionnaire items. Thus, data on epistemic beliefs collected in this study fitted the measurement model in the applied instrument.

To investigate changes in students’ epistemic beliefs between pre- and posttests, paired sample t-tests were conducted on the overall score of epistemic beliefs as well as the four dimensions. Results indicated that students’ overall epistemic beliefs had significantly improved, from 3.62 (SD = 0.31) to 3.90 (SD = 0.45), t(24) = 3.80, p < 0.001; so had scores in the dimensions of source of knowledge (t(24) = 2.61, p < 0.05) and justification of knowledge (t(24) = 2.96, p < 0.01). Improvement on the other two dimensions—certainty of knowledge (t(24) = 1.86, p = 0.08) and development of knowledge (t(24) = 1.72, p = 0.10)—was marginally significant (see Fig. 5). Note that higher scores in all scales represented more sophisticated beliefs, and students demonstrated significant growth in at least some dimensions of epistemic beliefs over a 10-week period.

Fig. 5
figure 5

Improvement of epistemic beliefs from pre- to posttest. Error bars represent standard errors. Points are offset horizontally to ensure visibility of all error bars

Full size image

Relationship among the conception of promisingness, epistemic beliefs and conceptual understanding

Correlation analysis was conducted among various measures, including the conception of promisingness, epistemic beliefs, and conceptual understanding, as well as their changes between pre- and posttests. Results indicated the conception of promisingness was significantly correlated with epistemic beliefs (r = 0.34, p < 0.05) and conceptual understanding (r = 0.29, p < 0.05). Also, growth in the conception of promisingness was significantly correlated with the growth in epistemic beliefs (r = 0.47, p < 0.05) and in conceptual understanding measured by the knowledge test (r = 0.52, p < 0.01). Therefore, even though causal relations cannot be confirmed with the present research design, the understanding of promisingness, epistemic beliefs and conceptual understanding appeared to develop in tandem.

Discussion and conclusions

Reflecting a common characteristic of design-based research (Edelson 2002), the present study had two major goals: (a) to understand the extent to which designed technological and pedagogical affordances could enable or augment the students’ capacity to engage in promisingness judgments, and (b) to explore how having elementary school students engage in promisingness judgments could promote scientific understanding and epistemic beliefs. The first goal maps onto the first research question, and the second goal covers the rest.

First, can elementary students take collective responsibility for choosing promising directions for their community knowledge building? Building on earlier iterations of a design-based research initiative on promisingness (Chen et al. 2011, 2015), this study continued to address this question. In particular, a refined version of the Promising Ideas Tool and pedagogical supports were applied in a class of sixth graders in Colombia who were studying biology. With pedagogical and technological supports, this class engaged in making promisingness judgments about their own community’s ideas over a 10-week period. Despite a school context favoring knowledge acquisition instead of progressive problem-solving, through classroom dialogues many students were able to depart from an initial focus on truthfulness and move towards a knowledge-building-potential conception of promisingness. Through the Review-Reflect-Refocus (3R) process centering on promisingness judgments, students sifted through a complex knowledge space for promising ideas, reflected upon their choices together, and collectively decided for themselves about which ideas and problems to pursue next. Analysis of both classroom dialogues and Knowledge Forum data demonstrated the progress that students achieved in understanding promisingness and making sensible promisingness judgments. This finding contributes to answering the question of whether students can assume higher levels agency in learning: “What seems to be required, in order for children successfully to assume executive control in their own zones of proximal development in a classroom setting, is a social process that allows the wisest judgments to work their way forward” (Scardamalia and Bereiter 1991, p. 58). Promisingness judgments represent such “wise judgments” in knowledge building. By engaging students in making promisingness judgments on their own, this study responds to the call for student agency and brings student work into greater alignment with real-world knowledge creation as well as the scientific practices advocated by the Next Generation Science Standards (National Research Council 2012).

Second, how did students’ conceptual understanding change during the intervention? This question is also important, as idea development is the central business of knowledge building. Results confirmed improvement in this area, reflected by both the pre-post tests comparison and the analysis of the scientific sophistication of ideas across discourse phases. The findings tended to be consistent with an earlier iteration of promisingness research (Chen et al. 2015). Unfortunately, no control class was available for comparison. Even though exploring the potential usefulness of designs instead of establishing causation was the focus of this study, more sophisticated research designs in the future could advance this research program by attending to causality.

How did students’ epistemic beliefs change during the intervention? Plausibly, individuals treating authoritative figures as the “owners” of knowledge are unlikely to make sensible promisingness judgments—simply because promisingness will not be an issue at all with such a mindset. In contrast, students who believe the development of scientific knowledge takes time and involves failures are more likely to tinker with their own ideas and look for promising paths to improve them. Results of the pre-post test comparison in the study indicated significant growth in epistemic beliefs over a relatively short period of time; students’ epistemic beliefs in the dimensions of source of knowledge and justification of knowing especially improved. This finding is significant given that more sophisticated beliefs are thought to emerge at the college level (e.g., Perry 1970), and changes with epistemic beliefs are typically examined across multiple years—in stead of weeks—in the literature (e.g., Cano 2005).

To what extent did these changes with conceptual knowledge, epistemic beliefs, and the understanding of promisingness correlate with each other? In classroom dialogues exploring the concept of promisingness, epistemic cognition (Chinn et al. 2011) was evident among students. The process of 3R provided opportunities for epistemic cognition, focused especially on the sources and justification of knowledge. The experience of determining the next steps for inquiry using promising ideas tagged by students themselves may have also helped overcome the epistemic vices (e.g., passiveness) that students brought with them into the study. It seems logical, therefore, that epistemic beliefs, the understanding of promisingness and conceptual knowledge were found to develop in tandem. Despite the lack of a control group, these findings pointed to potential fruitfulness of the promisingness intervention in facilitating students’ epistemic beliefs. The design-based research program has been focused so far on developing a robust intervention for promisingness judgments, treating as a whole the complex system involving multiple factors and emergent properties, such as cultural beliefs, curriculum, teacher, and technology (Bielaczyc 2006). This characteristic of design-based research has led to study results less focused on confirming the effectiveness of developed interventions—a goal that is sought by experimental paradigms aspiring for “harder” science but not necessarily desirable given the complex social settings where learning takes place (Phillips 2014). Future research is needed to explore possible causal linkages between promisingness judgments and epistemic beliefs, and more importantly, to engage with contexual factors and properties when facilitating student development in these areas.

This study advances the established line of knowledge building research. By engaging students in exploring the concept of promisingness and evaluating ideas with the Promising Ideas Tool, the study introduced new theoretical elements that remain otherwise largely unexplored in knowledge-building practices. The pedagogical designs explored in this study reinforce key principles of knowledge building (Scardamalia 2002; Chen and Hong 2016). For instance, the evaluation of promisingness is grounded on the principle of improvable ideas and idea diversity; it represents high-level epistemic agency. And emphasizing the role of knowledge-building discourse, the process of making promisingness judgments encourages collective responsibility and aims at rising above current community understanding. Future empirical and theoretical work on promisingness would help to further the advancement of knowledge building research and work on computer-supported collaborative learning (CSCL) in general.

In terms of technology, work on the Promising Ideas Tool could shed light on technological designs for online discourse in education. First, in the context of knowledge building, the tool extends Knowledge Forum by providing explicit sociotechnical supports for promisingness judgments. The tool essentially creates a “promisingness layer” above regular discourse for students to constantly contribute to, reflect upon, and work from. This promisingness layer could promote idea connectedness across boundaries and afford new mechanisms to move beyond current knowledge frontiers. Second, designs for promisingness judgments have highlighted new possibilities for nurturing self-organization and emergence in online discourse environments. The dynamics of 3R in this study showed promise in terms of using the results of promisingness judgments as traces of individual cognition to foster self-organization at the community level. Essentially, tagging promising ideas represents a community endeavor that leaves behind “stigmergy”—a mechanism of spontaneous, indirect coordination that keeps social insects coordinated. A promising direction for future technical development for promisingness is to support discussion and deliberation at scale on the Web, with promising ideas as stigmergy for self-organization within massive communities. Finally, this line of promisingness research has prompted efforts to seek automated indicators of promisingness by harnessing new advances in learning analytics. To facilitate idea improvement, besides engaging students in evaluating promisingness, we could also turn to computational techniques for additional insights to augment human judgments (e.g., Lee et al. 2016). For example, traces of multiple build-ons of one idea in Knowledge Forum may indicate some merit in terms of the promisingness of the idea; linguistics features in student-written text could provide additional indicators (Rosé et al. 2008). Future work could be done using learning analytics techniques to scaffold promisingness judgments and to facilitate continual idea improvement.

To conclude, this study implemented a design intervention in a class of sixth graders, with a goal of exploring the potential affordances of promisingness judgments for fostering conceptual understanding and epistemic growth. Results indicated that students were capable of improving their understanding of promisingness and making promisingness judgments deemed sensible by domain experts. Both conceptual understanding and epistemic beliefs of students improved over a 10-week period. Conceptual change, epistemic beliefs, and the understanding of promisingness were found to develop in tandem. This study contributes to the argument that students can take collective responsibility for their knowledge advances when provided suitable contexts (Zhang et al. 2007), and it points to future directions for theoretical, empirical, and technical developments in knowledge building and CSCL.