Keywords

1 Introduction

Knowledge representation is a key concept in psychological and educational diagnostics. Thus, numerous models for describing the fundamentals of knowledge representation have been applied so far. The distinction which has received the most attention is that between declarative (“knowing that”) and procedural (“knowing how”) forms of knowledge (see Anderson, 1983; Ryle, 1949). Declarative knowledge is defined as factual knowledge, whereas procedural knowledge is defined as the knowledge of specific functions and procedures for performing a complex process, task, or activity. Closely associated with these concepts is the term cognitive structure, also known as knowledge structure or structural knowledge (Jonassen, Beissner, & Yacci, 1993), which is conceived of as the manner in which an individual organizes the relationships between concepts in memory (Ifenthaler, Masduki, & Seel, 2009; Shavelson, 1972). Hence, an individual’s cognitive structure is made up of the interrelationships between concepts or facts and procedural elements.

Further, it is argued that the order in which information is retrieved from long-term memory will reflect in part the individual’s cognitive structure within and between concepts or domains. When compared to that of a novice, a domain expert’s cognitive structure is considered to be more tightly integrated and to have a greater number of linkages between interrelated concepts. There is thus immense interest on the part of researchers and educators to diagnose a novice’s cognitive structure and compare it with that of an expert in order to identify the most appropriate ways to bridge the gap (Ifenthaler, Masduki, et al., 2009; Ifenthaler & Seel, 2005). By diagnosing these structures precisely, even partially, the educator comes closer to influencing them through instructional settings and materials.

However, it is not possible to measure these internal representations of knowledge directly. Additionally, it is argued that different types of knowledge require different types of representations (Minsky, 1981). Therefore, we argue that it is necessary to identify economic, fast, reliable, and valid techniques to elicit and analyze cognitive structures (Ifenthaler, 2008a). In order to identify such techniques, one must be aware of the complex processes and interrelationships between internal and external representations of knowledge. Seel (1991, p. 17) describes the function of internal representation of knowledge by distinguishing three zones – the object zone W as part of the world, the knowledge zone K, and the zone of internal knowledge representation R. As shown in Fig. 12.1, there are two classes of functions: (1) fin as the function for the internal representation of the objects of the world (internalization) and (2) fout as the function for the external re-representation back to the world (externalization). Neither class of functions is directly observable. Hence, a measurement of cognitive structures is always biased as we are not able to more precisely define the above-described functions of internalization and externalization (Ifenthaler, 2008a). Additionally, the possibilities of externalization are limited to a few sets of sign and symbol systems (Seel, 1999b) – characterized as graphical and language-based approaches.

Fig. 12.1
figure 12_1_162453_1_En

Functions of representation and re-representation

Full size image

Lee and Nelson (2004) report various graphical forms of external representations for instructional uses and provide a conceptual framework for external representations of knowledge. Graphical forms of externalization include (1) knowledge maps, (2) diagrams, (3) pictures, (4) graphs, (5) charts, (6) matrices, (7) flowcharts, (8) organizers, and (9) trees. However, not all of these forms of externalization have been utilized for instruction and educational diagnosis (Ifenthaler, 2008a; Scaife & Rogers, 1996; Seel, 1999a). Other forms of graphical approaches are the structure formation technique (Scheele & Groeben, 1984), pathfinder networks (Schvaneveldt, 1990), mind tools (Jonassen, 2009; Jonassen & Cho, 2008), and causal diagrams (Al-Diban & Ifenthaler, in press).

Language-based approaches include thinking-aloud protocols (Ericsson & Simon, 1993), teach-back procedures (Mandl, Gruber, & Renkl, 1995), cognitive task analysis (Kirwan & Ainsworth, 1992), and computer linguistic techniques (Pirnay-Dummer, Ifenthaler, & Spector, 2009; Seel, Ifenthaler, & Pirnay-Dummer, 2009).

As discussed above, there are numerous approaches for eliciting knowledge for various diagnostic purposes. However, most approaches have not been tested for reliability and validity (Ifenthaler, 2008a; Seel, 1999a). Additionally, they are almost only applicable to single or small sets of data (Al-Diban & Ifenthaler, in press; Ifenthaler, 2008b). Hence, new approaches are required which have not only been tested for reliability and validity but also provide a fast and economic way of analyzing larger sets of data. Additionally, approaches for educational diagnostics also need to move beyond the perspective of correct and incorrect solutions. As we move into the twenty-first century, we argue that the application of alternative assessment and analysis strategies is inevitable for current educational diagnostics.

In this chapter, we focus on the scope of graphical indices in educational diagnostics. First, this chapter will provide an introduction to the implementation of graphs as external knowledge representations and present graphical indices and their possible applications in educational diagnostics. We will then highlight recent empirical studies which used graphical indices for educational diagnostics. The chapter will conclude with suggestions for future research for educational diagnostics using graphical indices.

2 Graphs as External Knowledge Representation

The underlying assumption is that knowledge can be re-represented (externalized) as a graph (Norman & Rumelhart, 1978). A graph consists of a set of vertices whose relationships are represented by a set of edges. The elements of a graph and their corresponding graphical measures are defined by the methods of graph theory (Diestel, 2000; Harary, 1974; Tittmann, 2003). Graph theory has been applied in various fields of research and applications, e.g., decision making, transactional analysis, network problems, transportation and traffic planning, scheduling problems, topology problems, and project management (see Chartrand, 1977). An overview of applications of graph theory in the social and psychological sciences has been provided by Durso and Coggins (1990) and in educational science by Nenninger (1980).

A widely accepted application of graph theory in social, educational, and psychological science is the use of Pathfinder networks (Schvaneveldt, 1990). Pathfinder provides a representation of knowledge by using pairwise similarity ratings among concepts to create a network. Pathfinder techniques have been combined with other procedures (e.g., multidimensional scaling – MDS) to expand the information for diagnostic purposes (e.g., Acton, Johnson, & Goldsmith, 1994; Goldsmith, Johnson, & Acton, 1991). However, Goldsmith et al. (1991) mention the need for more research regarding the psychological interpretation of graphs as knowledge representation. Accordingly, we argue that graph theory has potential beyond its application in the Pathfinder approach. The following sections will strengthen our assumptions.

2.1 Basics of Graph Theory

A graph is constructed from a set of vertices whose relationships are represented by edges. Basics of graph theory are necessary to describe externalized knowledge representations as graphs (Bonato, 1990; Ifenthaler, Masduki, et al., 2009).

  1. 1.

    A graph G(V,E) is composed of vertices V and edges E. If the relationship between vertices V is directional, a graph is called a directed graph or digraph D. A graph which contains no directions is called an undirected graph.

  2. 2.

    The position of vertices V and edges E on a graph G are examined with regard to their proximity to one another. Two vertices x, y of G are adjacent if they are joined by an edge e. Two edges e≠f are adjacent if they have a common end or vertex x.

  3. 3.

    A path P is a graph G where the vertices x i are all distinct. The length of a path P is calculated by the number of its edges e j . The vertices x 0 and x k are called the ends of the path P.

  4. 4.

    A graph G is indexed when single vertices V and edges E are distinguished by their names or content.

  5. 5.

    Every connected graph G contains a spanning tree. A spanning tree is acyclic and includes all vertices of G. Spanning trees can be used for numerous descriptions and calculations concerning the structure of a graph.

Please refer to Chapter 10, this volume, or to Tittmann (2003) for a detailed mathematical introduction to graphs and networks. The following part of this section will provide an overview of measures of graph theory which can be applied for educational diagnostics. However, as available measures of graph theory only account for structural properties of knowledge representations, the second to last part of this section will focus on measures beyond graph theory, namely semantic properties. The concluding part of this section will briefly describe the HIMATT tool, which integrates graphical indices for educational diagnostics (Pirnay-Dummer et al., 2009).

2.2 Measures of Graph Theory

By describing externalized knowledge representations as graphs, including associated vertices and edges, we are able to apply various measures from graph theory to diagnose individual knowledge representations and, in addition, to track the development of knowledge representations over time (Bonato, 1990; Ifenthaler, Masduki, et al., 2009; White, 1985). Below we briefly describe appropriate structural measures, including information on the (a) operationalization, (b) computation rules, and (c) diagnostic purpose of a knowledge representation. None of the structural measures account for the content of the underlying knowledge representation.

  1. 1.

    Number of vertices indicates the number of concepts (vertices) within a graph.

    1. a.

      The size of the knowledge representation is indicated by the sum of all embedded concepts (semantically correct or incorrect).

    2. b.

      Computed as the sum of all vertices within a cognitive structure. Defined as a value between 0 (no vertices) and N.

    3. c.

      The diagnostic purpose is to identify additions of vertices (growth of the graph) as compared to previous knowledge representations and track change over time.

  2. 2.

    Number of edges indicates the number of links (edges) within a graph.

    1. a.

      The size of the knowledge representation is indicated by the sum of all embedded links (semantically correct or incorrect).

    2. b.

      Computed as the sum of all edges within a cognitive structure. Defined as a value between 0 (no edges) and N.

    3. c.

      The diagnostic purpose is to identify additions of links (closeness of associations of the graph) as compared to previous knowledge representations and track change over time.

  3. 3.

    Connectedness indicates how closely the concepts and links of the graph are related to each other.

    1. a.

      The closeness of the knowledge representation is indicated by all possible paths and their accessibility.

    2. b.

      Computed as the possibility to reach every vertex from every other vertex in the knowledge representation. Defined as a value between 0 (not connected) and 1 (connected).

    3. c.

      The diagnostic purpose is to identify the strength of closeness of associations of the knowledge representation. A strongly connected knowledge representation could indicate a deeper subjective understanding of the underlying subject matter.

  4. 4.

    Ruggedness indicates whether non-linked vertices of a graph exist.

    1. a.

      The concepts of a knowledge representation are not accessible from every other concept. Hence, the knowledge representation consists of at least two subgraphs which are not linked.

    2. b.

      Computed as the sum of subgraphs which are independent or not linked. Defined as a value between 1 (all vertices are linked) and N.

    3. c.

      The diagnostic purpose is to identify possible non-linked concepts, subgraphs, or missing links within the knowledge representation. Non-linked concepts of a knowledge representation point to a lesser subjective understanding of the phenomenon in question.

  5. 5.

    Diameter indicates how large a graph is.

    1. a.

      The diameter of a knowledge representation is a reliable indicator for its complexity.

    2. b.

      Computed as the quantity of edges of the shortest path between the most distant vertices (diameter) of the spanning tree of a knowledge representation. Defined as a value between 0 (no edges) and N.

    3. c.

      The diagnostic purpose is to identify how broad the subject’s understanding of the underlying subject matter is.

  6. 6.

    Cyclic indicates the existence of paths within a graph returning back to the start vertex of the starting edge.

    1. a.

      A cyclic knowledge representation contains a path returning back to the start concept of the starting link.

    2. b.

      Computed as the existence or nonexistence of cycles within the knowledge representation. Defined as 0 (no cycles) and 1 (is cyclic).

    3. c.

      The diagnostic purpose is to identify the strength of closeness of associations of the knowledge representation. A cyclic knowledge representation could indicate a deeper subjective understanding of the underlying subject matter.

  7. 7.

    Number of Cycles indicates the number of cycles within a graph.

    1. a.

      A cyclic knowledge representation contains at least one path returning back to the start concept of the starting link.

    2. b.

      Computed as the sum of all cycles within a knowledge representation. Defined as a value between 0 (no cycles) and N.

    3. c.

      The diagnostic purpose is to identify the strength of closeness of associations of the knowledge representation. Many cycles within a knowledge representation could indicate a deeper subjective understanding of the underlying subject matter.

  8. 8.

    Average degree of vertices indicates the average degree of all incoming and outgoing edges of all vertices within a graph.

    1. a.

      An increase in the number of incoming and outgoing links adds to the complexity of the knowledge representation.

    2. b.

      Computed as the average degree of all incoming and outgoing edges of the knowledge representation. Defined as a value between 0 and N.

    3. c.

      The diagnostic purpose is to identify a low, medium, or high density within the knowledge representation. Knowledge representations which only connect pairs of concepts can be considered weak; a medium density is expected for most good working knowledge representations.

Hietaniemi (2008) offers a powerful open-source module called Graph-0.84 which includes the graph data structures and algorithms described above. The module can be implemented into PERL environments (The Perl Foundation, 2008). Features of the module have been implemented into the SMD Technology (Ifenthaler, 2008b), T-MITOCAR (Pirnay-Dummer, Ifenthaler, & Johnson, 2008), and HIMATT (Pirnay-Dummer et al., 2009).

2.3 Measures Beyond Graph Theory

Besides the measures of graph theory, which account for structural properties of knowledge representations, we argue that an educational diagnostics system should also account for the specific content (semantic properties). Therefore, we introduced semantic measures which add to the richness of detail of our proposed educational diagnostics (Johnson, Ifenthaler, Pirnay-Dummer, & Spector, 2009; Pirnay-Dummer et al., 2009). A semantic measure consists of a comparison feature which calculates similarities and contrasts between two or more different knowledge representations. Such measures for comparison can be applied to any knowledge representation which is available as a graph. Some of the measures count specific features of a given graph. For a given pair of frequencies f 1 and f 2, the similarity is generally derived by

$$s = 1 - \frac{{\left| f_1 - f_2 \right| }}{{\max (f_1 ,\, f_2 )}}$$

which results in a measure of 0 ≤ s ≤ 1, where s = 0 is complete exclusion and s = 1 is identity.

The other measures collect sets of properties from the graph (e.g., the vertices = concepts or the edges = relations). In this case, the Tversky similarity (Tversky, 1977) applies for the given sets A and B:

$$s = \frac{{f(A \cap B)}}{{f(A \cap B) + \alpha \cdot f(A - B) + \beta \cdot f(A - B)}}$$

α and β are weights for the difference quantities which separate A and B. They are usually equal (α = β = 0.5) when the sources of data are equal. However, they can be used to balance different sources systematically (e.g., comparing a learner model which was constructed within 5 min to an expert model, which may be an illustration of the result of a whole book).

So far, three semantic measures have been developed, implemented, and tested for reliability and validity: (1) concept matching, (2) propositional matching, and (3) balanced propositional matching. Below we briefly describe these three semantic measures, including information on their (a) operationalization, (b) computation rules, and (c) diagnostic purpose.

  1. (1)

    Concept matching compares the sets of concepts (vertices) within a graph to determine the use of terms.

    1. a.

      The use of semantically correct concepts (vertices) is a general indicator of an accurate understanding of the given subject domain.

    2. b.

      Computed as the sum of vertices of a knowledge representation which are semantically similar to a domain-specific reference representation (e.g., expert solution). Defined as a value between 0 (no semantic similar vertices) and N.

    3. c.

      The diagnostic purpose is to identify the correct use of specific concepts (e.g., technical terms). The absence of a great number of concepts indicates a less elaborated domain-specific knowledge representation.

  2. (2)

    Propositional matching compares only fully identical propositions between two graphs.

    1. a.

      The use of semantically correct propositions (vertex-edge-vertex) indicates a correct and deeper understanding of the given subject domain.

    2. b.

      Calculated as the semantic similarity between a cognitive structure and a domain-specific reference cognitive structure. Defined as a value between 0 (no similarity) and 1 (complete similarity).

    3. c.

      The diagnostic purpose is to identify the right use of specific propositions (concept-link-concept), i.e., concepts correctly related to each other. Additionally, misconceptions can be identified for a specific subject domain by comparing known misconceptions (as propositions) to individual knowledge representations.

  3. (3)

    Balanced propositional matching should be used instead of the concept and propositional matching to balance the dependency of both measures.

    1. a.

      Propositional matching necessarily has its maximum in the value of concept matching. In order to balance this dependency of both indices, the balanced propositional matching index should be used instead of the concept and propositional matching.

    2. b.

      Computed as the quotient of propositional matching and concept matching. Defined as a value between 0 (no similarity) and 1 (complete similarity).

    3. c.

      The diagnostic purpose is to account for the correct use of single concepts (e.g., technical terms) and their correct connectedness.

2.4 Implementation of Graphical Indices for Educational Diagnostics

The demand for an automated and computer-based diagnostic system incorporating a domain independent, fast, reliable, and valid assessment and analysis brought forth the HIMATT system (Highly Integrated Model Assessment Technology and Tools; see Pirnay-Dummer et al., 2009). Methodologically, the tools integrated into HIMATT touch the boundaries of qualitative and quantitative research methods and provide bridges between them. First of all, text can be analyzed very quickly without loosening the associative strength of natural language. Furthermore, concept maps can be analyzed and compared to those of an expert or other participant.

Figure 12.2 shows the architecture of HIMATT. Within the system, experiments can be laid out and conducted for various educational diagnostic purposes. Additionally, external data in written or graphical formats can be integrated into HIMATT. The data can then be analyzed by the researcher. As a result of the analysis process, HIMATT generates standardized graphical representations and seven quantitative indicators which are based on graph theory.

Fig. 12.2
figure 12_2_162453_1_En

HIMATT architecture

Full size image

Reliability measures exist for the individual instruments integrated into HIMATT. They range from r = 0.79 to r = 0.94 (Ifenthaler, 2008b; Pirnay-Dummer et al., 2009) and are tested for the semantic and structural measures separately and across different knowledge domains. Validity measures are also reported separately for the structural and semantic measures. Convergent validity lies between r = 0.71 and r = 0.91 for semantic comparison measures and between r = 0.48 and 0.79 for structural comparison measures (Pirnay-Dummer et al., 2009).

3 Empirical Studies

Empirical studies on the application of graph theory are available for almost every field of science. A literature review revealed over 14,000 scientific journal publications. The huge spectrum of research studies includes projects from management (e.g., Darvish, Yasaei, & Saeedi, 2009), geophysics (e.g., Todd, Toth, & Busa-Fekete, 2009), medicine (Chowdhury, Bhandarkar, Robinson, & Yu, 2009), engineering (e.g., Huang, Lo, Zhi, & Yuen, 2008; Rao & Padmanabhan, 2007), neuroscience (e.g., Bai, Qin, Tian, Dai, & Yang, 2009), physics (e.g., Ding & Guan, 2008), computer science (e.g., Bronevich & Meyer, 2008; Fiedler, 2007), biology (e.g., Ohtsuki, Pacheco, & Nowak, 2007), chemistry (e.g., Balaban, 1985), oceanography (e.g., Prigent, Fontenelle, Rochet, & Trenkel, 2008), and anthropology (e.g., Foster, 1978).

However, the number of empirical studies on the application of graph theory in the field of education is small (e.g., Durso & Coggins, 1990; Goldsmith et al., 1991; Hsia, Shie, & Chen, 2008; Nenninger, 1980; Schvaneveldt, 1990; Xenos & Papadopoulos, 2007). A series of empirical studies focusing on the application of graph theory in educational diagnostics using computer-based assessment and analysis techniques has been conducted recently. The graph theory-based analysis functions have been implemented into the HIMATT system (see above and Chapter 6 in this volume). In the following, we present three of these recent studies which provide insight into the possibilities of applying graph theory in educational diagnostics: (1) development of cognitive structures over time, (2) feedback for improving expert performance, and (3) between-domain distinguishing features of cognitive structures.

3.1 Development of Cognitive Structures

The study by Ifenthaler, Masduki et al. (2009) focuses on the issues involved in tracking the progression of cognitive structures, which captures the transition of learners from the initial state to the desired state (Snow, 1989, 1990), and making repetitive measurements of change over an extended period of time for a more accurate diagnosis (Ifenthaler & Seel, 2005; Seel, 1999a). Accordingly, it responds to the claim that research on cognitive structures needs to move beyond the traditional two-wave design in order to capture changes more precisely (Willett, 1988). As individuals reinstate and modify their cognitive structures when interacting with the environment (Jonassen et al., 1993, Piaget, 1976; Seel, 1991), the necessity of conducting multiwave longitudinal experiments is evident. However, the collection and analysis of longitudinal data gives rise to various methodological dilemmas which should not be neglected (see Ifenthaler, 2008a; Seel, 1999a). Besides general concerns about quantitative studies over time (Collins & Sayer, 2001; Moskowitz & Hershberger, 2002), tracking changes in cognitive structures requires valid and reliable assessment techniques, adequate statistical procedures, and specific situations which enable the activation of such cognitive structures (Ifenthaler, 2008a).

Indicators of graph theory have been assumed to be applicable for tracking the development of externalized cognitive structures over time.

Twenty-five students (18 female and 7 male) from the University of Freiburg, Germany, participated in the study. Their average age was 24.7 years (SD = 1.9). All students attended an introductory course on research methods in winter semester 2007. A total of 125 concept maps were collected at five measurement points during the semester.

Data were collected through concept maps using the software CmapTools (Cañas et al., 2004). According to Novak (1998), a concept map is a two-dimensional graphical representation of communicated knowledge and its underlying structure. A concept map consists of concepts (graph theory: vertices) and relations (graph theory: edges). Research studies on the application of CmapTools indicate that our theoretical assumptions on using this software are widely accepted (e.g., Coffey et al., 2003; Derbentseva, Safayeni, & Cañas, 2004). Since the research study focused on the development of cognitive structures, the longitudinal procedure included five measurement points. The main parts of the study were as follows:

  1. (1)

    In a 60-min introductory lesson, the subjects were introduced to the concept mapping technique and taught how to use the CmapTools software. Additionally, the instructor collected demographic data and delivered documentation on concept maps and the software, including examples.

  2. (2)

    At five measurement points (MP) during the course on research methods, the subjects were asked to create an open concept map relating to their understanding of research skills. Every subject needed to upload the concept map at a specified date and time during the course.

  3. (3)

    The course learning outcome was measured by way of (1) five written assignments, (2) a written exam, and (3) a written research proposal. The course learning outcome was rated with a score between 0 and 100 points (Spearman-Brown coefficient, r = 0.902).

After uploading the concept maps, the instructor gave the students a brief feedback to notify them that their maps had been successfully uploaded and that they should carry on with their studies in the course. As open concept maps were used in the research study, the subjects were not limited to specific words while annotating the concepts and relations.

An in-depth analysis of N = 125 cognitive structures (five re-representations of each of the 25 participants) revealed several patterns that helped us to better understand the construction and development of these constructs over time. Several HLM analyses were computed to test the hypothesis. According to the guidelines suggested by Hox (2002), the sample size of the study is just adequate. However, in order to validate the initial findings further studies with a larger sample size will be necessary.

The results of the Level-1 HLM analysis (intraindividual change of cognitive structures over time) are described in Tables 12.1 and 12.2. The Mean Initial Status π0i indicates that all corresponding measures are significantly higher than 0. Except for average degree of vertices, all measures reveal a significant positive linear Mean Growth Rate π1i per measurement point (e.g., surface structure = 15.36).

Table 12. 1 Level-1 linear growth models of cognitive structures (structural measures)
Full size table
Table 12. 2 Level-1 linear growth models of cognitive structures (semantic measures)
Full size table

Therefore, H1.1 can be accepted: The structure (surface structure, graphical structure, ruggedness, number of cycles, and number of vertices) of the externalized cognitive structures changes during the learning process, except for the measure average degree of vertices. The average degree of vertices indicates the average number of incoming and outgoing edges. Accordingly, as most of the externalized cognitive structures are very broad and do not center on one vertex, each vertex takes two edges on average. This does not change during the learning process, as the subject domain (research skills) does not change and does not seem to be organized around one central vertex.

The result of the HLM analysis revealed a significant growth in the structural measures between measurement points one and five. The overall size of the cognitive structures (surface structure) increased many times over. This is an indicator for an accommodation process (see Piaget, 1976; Seel, 1991), i.e., the individuals continuously added new concepts and links between concepts to their cognitive structures while learning. As a consequence, the complexity of the externalized cognitive structures also increased, which is indicated by the growth in the measures matching structure and number of cycles. Therefore, we conclude that in the process of learning and understanding more and more about a given subject matter, individuals succeed in integrating single concepts and links more tightly. However, we also found significant growth in the measure ruggedness (i.e., non-linked concepts within the entire cognitive structure). The significant decrease in the measure connectedness supports this result. This indicates that newly learned concepts are not immediately integrated into the cognitive structure. The delay involved in integrating concepts into the cognitive structure should be kept in mind when constructing instructional materials and learning environments. We also suggest analyzing this phenomenon more precisely in a future study.

Contrary to the results of the structural measures, the HLM analysis revealed significant growth only in the semantic measure vertex matching. The individuals use more and more semantically correct concepts (vertices) in the course of the learning process. As individuals become more familiar with the terminology of the subject domain, they use these concepts more frequently. This learning process enables individuals to communicate their cognitive structures more precisely and in a more expert-like manner. The significant positive correlation we found between the final learning outcomes and the number of semantically correct concepts (vertex matching) reaffirms these assumptions (see Ifenthaler, Masduki et al., 2009).

Hence, in order to provide effective instruction, it is important for students’ prior knowledge to be identified since the subsequent construction and organization of knowledge structures as well as mental models in a particular situation depends on the students’ preconceptions and naïve theories (Seel, 1999a). Measures derived from graph theory proved to be reliable and valid indicators.

3.2 Feedback for Improving Expert Performance

In this chapter, Ifenthaler (in press-b) investigates different types of model-based feedback generated automatically with the HIMATT (Highly Integrated Model Assessment Technology and Tools) methodology (see Pirnay-Dummer et al., 2009). Since the beginnings of mental model research (e.g., Gentner & Stevens, 1983; Johnson-Laird, 1983; Seel, 1991) many research studies have provided evidence that “mental models guide and regulate all human perceptions of the physical and social world” (Seel & Dinter, 1995, p. 5). Accordingly, mental models are dynamic ad hoc constructions which provide subjectively plausible explanations on the basis of restricted domain-specific information (Ifenthaler, 2008b). Various research studies have shown that it is very difficult but possible to influence such subjectively plausible mental models by providing specific information (see Anzai & Yokoyama, 1984; Ifenthaler & Seel, 2005; Mayer, 1989; Seel, 1995; Seel & Dinter, 1995). Ifenthaler and Seel (2005) argue that it is important to consider how such information is provided to the learner at specific times during the learning process and how it is structured. In accordance with the general definition of feedback (see Wagner & Wagner, 1985), such information for improving individual mental model building processes provided purposely and on the fly is referred to as model-based feedback.

Hence, model-based feedback aims at a restructuring of the underlying representations and a reconceptualization of the related concepts and links (vertices and edges). This is in following with Piaget’s epistemology (1950, 1976). New information provided via model-based feedback can be assimilated through the activation of an existing schema, adjustment by accretion, or tuning of an existing schema. Otherwise it is accommodated by means of a reorganization process which involves building new mental models (Seel et al., 2009).

Seventy-four students (66 female and 8 male) from the University of Freiburg, Germany, participated in the study. Their average age was 21.9 years (SD = 2.3). The participants were randomly assigned to the three experimental groups: (1) cutaway feedback (n = 26), (2) discrepancy feedback (n = 24), and (3) expert feedback (n = 24).

First, the participants completed a demographic data questionnaire. Second, they completed the concept map and causal diagram experience questionnaire. Next, the participants completed the test on verbal (6 min) and spatial abilities (9 min). Then they answered the 27 multiple choice questions of the domain-specific knowledge test on climate change (pretest). After a short relaxation phase, the participants were given an introduction to concept maps and causal diagrams and were shown how to use the HIMATT software. Then, the participants used the username and password they had been assigned to log in to the HIMATT system, where they constructed a causal diagram on their understanding of climate change (10 min). Immediately afterward, they wrote a text about their understanding of climate change (10 min). A short relaxation phase followed, during which we automatically generated the individual feedback models for each participant. After that, the participants received the text on climate change and the automatically generated feedback model (cutaway, discrepancy, or expert model – depending on which experimental group they had been assigned to). All three types of feedback models were automatically generated with HIMATT. The cutaway feedback model included all propositions (vertex-edge-vertex) of the participants’ pretest causal diagram. Additionally, the semantically correct vertices (compared to the expert re-representation) were graphi-cally highlighted (circles are semantically correct as compared to the expert; ellipsis are semantically incorrect as compared to the expert re-representation). The discrepancy feedback model included only propositions (vertex-edge-vertex) of the participants’ pretest causal diagram which had no semantic similarity to the expert re-representation. The expert feedback model consisted of a standardized re-representation of an expert’s model on climate change. The participants had 15 min to read the text and examine their feedback model. Immediately after working on the text, the participants completed the model feedback quality test. Then they answered the 27 multiple choice questions of the posttest on declarative knowledge. After another short relaxation phase, the participants used their username and password to log in to the HIMATT system for the second time. In the HIMATT posttest, they constructed a second causal diagram on their understanding of climate change (10 min) and wrote a second text regarding their understanding of climate change (10 min). Finally, the participants had to complete a short usability test regarding HIMATT features.

The graphical re-representations of the participants were analyzed automatically with the HIMATT analysis feature. Hence, the knowledge gain of the seven HIMATT measures was computed by subtracting the pre- from the post-measure. Table 12.3 shows the average gain of the HIMATT measures (surface, graphical, structural, gamma, concept, propositional, and balanced propositional matching) for the three experimental groups (cutaway feedback, discrepancy feedback, and expert feedback).

Table 12.3 Average gain of HIMATT measures for the experimental groups (N = 74)
Full size table

The results showed a significant effect between participants in the three experimental groups for the HIMATT measure surface matching, F(2, 70) = 4.080, p = 0.021, η 2 = 0.10, with participants of the expert feedback group increasing their number of vertices more than the other experimental groups. The one-way ANOVA also revealed a significant effect for the HIMATT measure graphical matching, F(2, 70) = 7.355, p = 0.001, η 2 = 0.17. The increase of complexity of participants was higher in the expert feedback group than in the others. Between the experimental groups, the increase of the HIMATT measure structural matching was also significant, F(2, 70) = 3.140, p = 0.049, η 2 = 0.08. Again, the participants in the expert feedback group outperformed the other experimental groups. For the semantic HIMATT measure concept matching a final significant effect was found, F(2, 70) = 3.243, p = 0.045, η 2 = 0.08. Here, participants in the cutaway feedback group gained more correct concepts than the participants in the other two groups. However, no further effects for the HIMATT measures were found.

With the help of the seven automatically calculated HIMATT measures, changes in the participants’ understanding of the subject domain “climate change” were investigated and re-represented with causal diagrams. Participants who received the expert feedback added significantly more relations to their causal diagrams (surface matching) than did those in the other groups. Accordingly, the expert feedback provided them a broad spectrum of concepts and relations, which were then integrated into their own understanding of the phenomenon in question. This also explains the significant differences between the measures graphical and structural matching. As the number of relations in a causal diagram increases, there is a high probability that its complexity and complete structure will also increase.

However, an increase in these structural measures does not necessarily mean that the solutions of participants in the expert feedback group are better than these of the other participants. As a further analysis of the semantic HIMATT measures revealed, participants in the cutaway feedback group outperformed the other parti-cipants with regard to their semantic understanding of the phenomenon in question (concept matching). Accordingly, even if the structure increases, the semantic correctness of the learner will not automatically also increase. Hence, learners may integrate a huge amount of concepts into their understanding of the phenomenon which do not necessarily help them to come to a better and more correct solution to the problem.

Thus, measures derived from graph theory also proved to be reliable and valid indicators in the study on model-based feedback. Further studies will focus on the learning trajectories while providing forms of model-based feedback. This will provide more detailed insight into the effects of model-based feedback and how it helps to support and improve expertise and expert performance.

3.3 Between-Domain Distinguishing Features of Cognitive Structures

In this study, Ifenthaler and Hetterich (under review) argue that previous empirical studies have focused on within-domain-specific features and the learning-dependent development of cognitive structures (e.g., Clariana & Wallace, 2007; Ifenthaler, Masduki et al., 2009; Koubek, Clarkston, & Calvez, 1994). In contrast to these empirical investigations, this study focuses on between-domain specific similarities and differences. More precisely, it identifies similarities and differences in externalized cognitive structures between three different subject domains: mathematics, biology, and history.

The central research objective was to identify between-domain distinguishing features of externalized cognitive structures. Accordingly, the participants were asked to externalize their understanding of three different subject domains (mathematics, biology, history). Additionally, it is argued that the form of externalization influences the person’s communicated output (Ifenthaler, 2008a). Therefore, the participants were asked to externalize their understanding of each subject domain as written text and as a concept map.

Seventy-one students (66 female and 8 male) from the University of Freiburg, Germany, participated in the study. Their average age was 22.2 years (SD = 2.3). First, the participants completed a demographic data questionnaire and the experience with concept mapping test. Second, they completed the test on verbal, mathematical, and spatial abilities. Next, they were given an introduction to concept maps and causal diagrams and were shown how to use the HIMATT software (Pirnay-Dummer et al., 2009). After a short relaxation phase, the participants completed the domain-specific knowledge test on history. Then they received the text on European borders. The participants had 15 min to read the text. Then, they logged into the HIMATT system, where they constructed a causal diagram on their understanding of European borders (10 min). Immediately afterward, they wrote a text about their understanding of European borders (10 min). After another short relaxation phase, the procedure was repeated with the domains mathematics and biology ((1) Domain-specific knowledge test, (2) reading the text, (3) constructing a concept map, and (4) writing a test). In total, the experiment took approximately 2 h.

Overall, we found highly significant differences in the four structural measures of HIMATT between the three subject domains – for both written text (Table 12.4) and concept maps (Table 12.5). The ANOVA revealed a significant effect for written text for the measures graphical matching (F (2, 208) = 3.064, p < 0.05; η 2 = 0.03) and gamma matching (F (2, 208) = 8.929, p < 0.001; η 2 = 0.08).

Table 12.4 Means, standard deviations of the four structural measures of HIMATT for the written text
Full size table
Table 12.5 Means, standard deviations of the four structural measures of HIMATT for the concept maps
Full size table

For the concept maps, the ANOVA revealed a different picture. A significant effect was found between the three subject domains for the measures surface matching (F (2, 207) = 25.271, p < 0.001; η 2 = 0.20), graphical matching (F (2, 207) = 8.186, p < 0.001; η 2 = 0.07), and structural matching (F (2, 207) = 36.540, p < 0.001; η 2 = 0.26).

The findings indicate that there are similarities and differences between the structural features of the externalized cognitive structures. Additionally, initial analysis also indicates similarities and differences between the two forms of externalization (written text and concept map). This new research on the application of graph theory measures in educational diagnostics indicates another useful application of these quantitative indices.

4 Conclusion

There is an immense field of applications for graphical indices in educational diagnostics. Graph theory has proven to be an appropriate diagnostic approach, especially in knowledge representation and analysis. Pathfinder and combined techniques (Durso & Coggins, 1990; Schvaneveldt, 1990) provide a reliable representation of knowledge structures and analysis of learning as they use pairwise similarity ratings among concepts to create networks. These networks are based on proximity data among entities and are determined by calculating the proximities that best fit within the network. Furthermore, newly developed automated applications such as SMD Technology (Ifenthaler, 2008b), T-MITOCAR (Johnson et al., 2009; Pirnay-Dummer et al., 2008), and HIMATT (Pirnay-Dummer et al., 2009) integrate the latest software technology and a great quantity of graph theory-based applications and analysis functions.

Additionally, graph theory can be applied to almost every area of educational diagnostics. Picard (1980) introduced a very promising approach for designing and analyzing questionnaires using graph theory. Furthermore, graph theory has been successfully applied for instructional planning (Hsia et al., 2008) and evaluation purposes (Xenos & Papadopoulos, 2007).

Future applications of graph theory in educational diagnostics include automated self-assessment and forms of automated feedback. A recently implemented application is TASA (Text-Guided Automated Self-Assessment). TASA is a web-based online tool for self-assessment of written essays. It embeds the parts of SMD Technology (Ifenthaler, 2008b) and T-MITOCAR (Johnson et al., 2009) which are necessary to generate a graph from the learners’ essay directly after the upload. The uploaded essay provides the learner with a graphical representation of the essay in a format which non-experts have been shown to be able to handle. Additionally, graph theory-based measures make TASA into both a reflection and a preflection tool for the learner: After the upload is finished, the learners receive written feedback on the text. The text provides information on the key concepts, the ways in which they are connected, and concepts and connections which may be circumstantial but still have some added meaning in the text. TASA uses measures from graph theory to generate this feedback. If there is a group of learners which is working on the same task or topic, TASA may also be used as a preflection tool. Preflection will allow the learners to plan their actions based on what is already there and the task (goal) to fulfill. Once all members of the group have uploaded their text, TASA generates a list of the most common terms from all texts throughout the group. The learners are then asked which five terms from the whole list they would like to have in their underlying model (knowledge representation) when they upload their essay the next time. They select from a list of 30 terms. In this way, the individual learner can benefit from the other learners’ conceptions.

In our digital age, technology, learning, and educational diagnostics are closely linked (Ifenthaler, in press-a; Ifenthaler, Isaias, Spector, Kinshuk, & Sampson, 2009). Researchers and engineers have always endeavored to design and develop useful diagnostic systems to serve professional communities in the field of learning and instruction, and they will continue to do so (Ifenthaler, in press-a). Future work on automated computational diagnostics, including approaches such as graph theory, will provide more and more powerful dynamic systems for the comprehensive analysis of large amounts of data in a short space of time.