1 Introduction and Description of the Emerging Technology

There are several existing automatic writing evaluation (AWE) software tools for supporting free-text responses in online courses. However, the existing AWE tools are typically used for summative purposes (e.g., scoring), based on linguistic assessment (e.g., grammar), and/or implemented only in a single language. We have developed a formative structural AWE tool that can be used for multiple language text analysis, called Graphical Interface of Knowledge Structure (GIKS), that can immediately convert students’ writings into visually represented knowledge structure (KS) network graphs, a visual and concise snapshot of the degree to which the learners have organized and comprehended the content.

GIKS system has been developed by integrating two standalone offline software tools from different research traditions, ALA-Reader (Clariana and Wallace 2007) from education and Pathfinder KNOT (Schvaneveldt 2004) from a graph-theoretic cognitive science approach. Simply put, the ALA-Reader software tool captures the sequence of preselected important key terms in a text (e.g., student’s essay) as pair-wise links in a proximity file (see Kim 2012, for the validity of ALA-Reader). However, ALA-Reader tends to over-specify term links, thus adding some error to the data (Clariana et al. 2009). Pathfinder KNOT is a computer-based “data reduction” network scaling technique that searches for the strongest link, or most direct path, between the nodes (i.e., eliminating much of the complexity, or visual noise, of the original network; see Sarwar 2011, for the validity of KNOT). For this reason, ALA-Reader was designed specifically to output file formats that are directly readable by Pathfinder KNOT; Pathfinder algorithm is applied to convert the raw proximity data from ALA-Reader into a Pathfinder Network (PFnet) so that the resulting PFnet can represent the most salient linkages between co-occurring key terms in the text, or KS (see Fig. 1).

Fig. 1
figure 1

Fully connected PFnet from ALA-Reader (top) and reduced PFnet from ALA-Reader + Pathfinder KNOT (bottom) derived from the lesson text, Communications through Infrasound

Full size image

Notably, the ALA-Reader + Pathfinder (now GIKS) approach is not language-dependent because it uses pattern matching to capture the relations of preselected keywords in a text, so it can be applied in other language contexts beyond English (see Fig. 2 for example).

Fig. 2
figure 2

An example of a PFnet from Korean (top right), Arabic (top left), and Chinese (bottom), on the same lesson text, Communications through Infrasound

Full size image

2 Relevance for Learning, Instruction, and Assessment

It is becoming widely understood that meaningful knowledge leads to well-organized knowledge (e.g., novice-to-expert, Herr 2009). For example, the National Research Council recommends assessing KS as a key indication of meaningful learning (NRC 2000). Learning and attaining new knowledge is mainly dependent on how we organize the knowledge in our memory (Clariana 2010). Similarly, Spector and Koszalka (2004) argued that one could view learning progress as cognitive changes in the direction of expert-like KS. In this sense, learning progress is likely to be well-illustrated by the notion of KS when learning progress is characterized as a set of directional changes in a learner’s KS (Ifenthaler and Pirnay-Dummer 2014; Schlomske and Pirnay-Dummer 2009). Thus, we propose that KS is worth measuring.

Then, how can we effectively capture and visually represent KS? Recent cognitive studies emphasize KS as associative networks of concepts with weighted connections (much like a mental lexicon that consists of weighted associations between concepts, see Ifenthaler 2014, for detail), and learning can facilitate the strengthening of link weight as well as the enrichment of the concepts in the mental network (Meyer et al. 2012). Clariana et al. (2013, 2014) argued that learning outcomes should be examined with the framework of a nomological network in which the interrelationships among constructs are identified and empirically tested, and thus methods that can capture network properties can be the most effective for representing and describing the KS. Based on this connectionist theoretical bias, ALA-Reader + Pathfinder approach was developed and has been used to capture, visually represent, and compare KS as network graphs inherent in texts, because writing closely resembles thinking and learning and is thus a concrete artifact or manifestation of cognition (Emig 1977).

Since its initial operation by Clariana (2004), the ALA-Reader approach has been modified and improved over time through a number of investigations in diverse domains within and across languages to measure KS from texts - of monolingual speakers of English (Clariana et al. 2014), German (Clariana et al. 2013), and Dutch (Fesel et al. 2015), and also of bilingual speakers of Dutch/English (Mun 2015), Chinese/English (Tang and Clariana 2016), and Korean/English (Kim and Clariana 2015, 2016).

Also, the ALA-Reader approach has been favorably compared to other computational text analysis models, including global-scale (implicit) models such as Latent Semantic Analysis and Hyperspace Analogue of Language (LSA and HAL; e.g., Su and Hung 2010) and also local-scale (explicit) models such as T-MITOCAR and CMM (e.g., Kim 2012). Overall, these investigations reported similar or better performances of ALA-Reader for scoring essays. For example, in Su and Hung (2010) study, students’ essays were scored with LSA, a commercial essay scoring system, and ALA-Reader and each derived essay scores were compared to human raters. The correlation obtained were: LSA to raters, r = .56 and ALA-Reader to raters, r = .71, ALA-Reader outperformed LSA. Distinctively, the LSA is a ‘black box’, users (e.g., students) don’t know what terms are involved nor how it derived its structures, only providing a numeric score—it does not focus on a particular misconception a student has; whereas the ALA-Reader approach is a ‘glass box’ in that it provides a visual display of KS, allowing quick visual examination and comparison among KS representations—reflection on KS has repeatedly been shown to be an effective means for formative assessment (e.g., Sarwar and Trumpower 2015).

3 GIKS in Practice

Despite the large amount of ALA-Reader applications to measure KS in a text, there has been very few studies examining the effectiviness of KS network graphs in online learnig contexts, mainly because ALA-Reader + Pathfinder procedure is not currently automated. Thus, we have recently developed a browser-based online version of the ALA-Reader + Pathfinder, GIKS with support from the Pennsylvania State University's Center for Online Innovation in Learning (COIL). If individual students read a lesseon text and submit their writing responses (e.g., summary essay) on the GIKS website, then the software will immediately generate and display KS network graphs of their writings and of the lesson text (or a referent benchmark KS) to indicate specific areas of students’ current knowledge strengths and weaknesses (see Fig. 3). For example, based on the student’s KS shown in Fig. 3, she may note that she did not use the key concepts, talk, benefit, low-pitched, and scientists, and where these key concepts may fit and that she associated distance with things and animals rather than with access, i.e., this feedback could support “self-regulated learning” by providing a visual knowledge artifact for reflection—namely, through reflection on their KS in comparison with a referent KS. Reflection on feedback regarding the structure of one’s own knowledge can empower students by making learning engaging and personal (Clariana 2010); reflection can be done well or badly, successfully or unsuccessfully, but it is always a productive learning experience (Spector and Koszalka 2004). Thus, the GIKS system can promote online students’ active engagement in the development of their KS during learning by providing individualized KS feedback, and also benefit instructor’s understanding of their students’ KS and thinking, which may lead to using improved pedagogy and individualized instructional strategies.

Fig. 3
figure 3

GIKS screen shot of a KS network graph derived from a lesson text (left), Communications through Infrasound, and a student’s summary essay (right). Student’s KS feedback consists of a highlighted network graph showing the similarity and difference compared to the referent KS; yellow indicates ‘missing’ links/nodes and red indicates ‘incorrect’ links/nodes. *GIKS is open to public with invitation code at http://coil-giks.rhcloud.com/. (Color figure online)

Full size image

4 Significant Challenges and Conclusions

One significant challenge arose in our experiments of ALA-Reader is that this approach may extend only to analysis of texts that include fairly technical and thus specific vocabulary. Less technical content will likely include term synonyms and metonyms across texts that may not be recognized by the ALA-Reader approach, and this would certainly add error in the analysis in proportion to how many of these alternate forms of the selected terms are missed by the software (Clariana 2010). This increased error can be mitigated to some degree by careful identification of synonyms and metonyms that must then be included in the analysis. Nevertheless, additional research is needed to determine the validity of the ALA-Reader approach for less technical text content (e.g., narrative text).