Introduction

To integrate technology into classroom teaching and learning, the notion of technological pedagogical content knowledge (TPACK) has been developed since 2003 (Lundeberg et al., 2003; Mishra & Koehler, 2006). The term of design thinking has been often employed in teacher education not only to build teacher creativity and empathy but also to foster twenty-first century competencies of students as knowledge workers in a knowledge-intensive society (Koh et al., 2015). However, TPACK and design thinking still need to be further understood in the era of digitally-mediated learning generating the large amount of multimodal data including both unstructured data and structured data designed in digitally-mediated learning (Siemens, 2012) incidental to that learning.

Recently, to support a paradigm shift for twenty-first century learning, teacher design work has gained increasing attention by re-conceptualizing teachers as designers rather than curriculum deliverers (McKenney et al., 2015). The lack of teacher knowledge, skills, abilities, or competencies related to innovative learning environments in teaching have been identified as the major barriers to student design work (Kim, 2019). In other words, while teacher learning and teacher design work to support student design work in a digitally-mediated learning environment are of growing importance, very few teachers have the competencies and the confidence in their abilities to become designers (Kali et al., 2015). Further, despite the efforts to conceptualize teaching as a design science (Laurillard, 2012), assessing teacher design work can be challenging given that there are very few research tools to assess teacher design knowledge (TDK) competencies. To address this issue, this study proposes a survey that assesses TDK competencies in the era of digitally-mediated learning.

Identification of teacher design knowledge competencies dimensions

Recently a systematic review conducted by Kim (2019) defined teacher design knowledge (TDK) competencies for STEM teachers to make TDK visible in the form of TDK competency taxonomies. By reviewing existing teacher design work in technology-enhanced learning environments, these TDK competency taxonomies consist of four main categories: data practice, design practice, knowledge creation practice, and professional teaching practice (see Table 1, p. 4). These TDK competencies were used as the identified constructs to create the questionnaire item pool for the TDK competencies survey.

Table 1 Teacher design knowledge (TDK) competency taxonomy
Full size table

Data practice is defined as “documenting, creating, manipulating, and visualizing student learning data produced, shared, and improved by the community of learners across formal and informal settings” (p. 5) using technologies. In a digitally-mediated learning environment, compared to a traditional classroom setting, a large amount of multimodal learning artifacts (e.g., digital storytelling, drawings, photos, construction, games, scientific models) as epistemic artifacts (i.e., ideas See Bereiter, 2002) at cognitive, metacognitive, and socioemotional levels (D'Mello et al., 2014; Mega et al., 2014) is produced, shared, and improved individually and collectively. Also, creating data is important because not all data already produced, so teachers use technological tools (e.g., simulations, games, virtual labs) by collaborating with researchers and practitioners to allow students to generate data which are not accessible to our sense. Teachers are expected to manage, organize, examine, and analyze multiple sources of these data in meaningful ways to make sense and visualize student learning. Thus, teachers need to include these multimodal learning data to make instructional decisions and inform teaching practice beyond assessment results as the common form of data (Mandinach et al., 2015). A total of 19 items were written down in this data practice.

While documenting, creating, analyzing, and visualizing student learning data, teachers are expected to continually develop and improve their understanding of the nature and process of the design of learning environments. By drawing upon a Learning-by-Design model for a pedagogy of multiliteracies initiated by Cope and Kalantzis (2009), design is viewed as a meaning-making activity that enables transformation of resources of the already designed world of representation (so-called “Available Designs”). In the act of Designing of meaning making, the teacher appropriates, revoices, and transforms Available Designs and “enacts a new design” (p. 177). In particular, in order to synthesize emerging design knowledge about the use of digital technologies in designing learning environments, it is recommended for teachers to improve educational practices through design-enactment-refinement iterations (Wang & Hannafin, 2005) and create contextually-sensitive design principles. Thus, in design practice, it is essential for teachers to conduct rigorous and reflective inquiry to (a) appropriate known design principles through learning, (b) apply known and hypothetical design principles to address complex problems in real contexts in collaboration with practitioners and researchers, (c) test and define (improvise) new design principles to design innovative learning environments (in new designs), and (d) redefine design principles in multiple iterations (Kim, 2019, p. 8). A total of 17 items were written down in this design practice.

In line with the design practice, the teacher as “the meaning-maker-as-designer” (Cope & Kalantzis, 2009, p. 177) is expected to teach the students of generation ‘P’ (participatory, Jenkins, 2006 as cited in Cope & Kalantzis, 2016c) as active knowledge-makers rather than passive recipients in the knowledge society where knowledge is no longer viewed as an individual static entity (Valsiner & Veer, 2000; Van Aalst, 2009). Teachers need to focus more on supporting students in participating in various phases of the design as a form of action in knowledge creation practice where students become as epistemic agents by generating and identifying promising ideas and continually improving the ideas through sustained collaborative inquiry (Bereiter, 2010) and collective cognition (Chan, 2013). In this knowledge creation practice, there are four knowledge practice taxonomies in alignment with the above-mentioned data practice and design practice: (a) epistemic agency for an idea-centered discourse, (b) distributed social interactions for collective knowledge advancement, (c) collaborative formative assessment for knowledge creation, and (d) expansive learning for knowledge transformation. A total of 18 items were written down in this knowledge creation practice.

Although in the creative economy and knowledge society, the concept and practice of ‘teacher professionalism’ has gained increasing attention, there remains an uncertainty of what comprises “professional” teaching. The concept of teacher professionalism may evoke many images and have multiple definitions depending on the educational context, but in alignment with the above-mentioned data practice, design practice, and knowledge practice, four taxonomies were defined in teacher professionalism (Kim, 2019): (a) Adaptability, (b) Teacher Collaboration, (c) Design Thinking, and (d) Teacher Leadership (p. 17). A total of 23 items were written down in this professional teaching practice.

Research method

The purpose in developing these TDK competencies was to define and determine a teacher’s knowledge, skills, and attitudes of TDK according to a literature review of teacher design work in diverse settings (see Kim, 2019). Since the TDK competencies were developed based on the views and experiences of teachers who were teaching in diverse subjects and settings, the item pool to develop the scale was created by using all of the TDK competencies. A total of 16 competencies were listed within 4 competency domains (data, design, knowledge creation, professional teaching) (see Table 1 above). In order to determine the content validity of the scale form prepared, the expert teachers were asked for their views prior to the implementation process. For this purpose, the TDK competencies scale form was presented to a total of 3 experts (two expert teachers, one curriculum developer). Based on the feedback from those experts, items were revised, simplified and broken down into specific items.

As a result of this step, the scale form included a total of 77 items in 4 dimensions with 4 sub-dimensions for each dimension. There were 19 items in the dimension of data practice (see Table 2), 17 items in the dimension of design practice (see Table 3), 18 items in the dimension of knowledge creation practice (see Table 4), and 23 items in the dimension of professional teaching practice (see Table 5). For each item, five-point Likert type scale (“strongly agree”, “somewhat agree”, “neither agree nor disagree”, “somewhat disagree”, and “strongly disagree”) was used. The final version of the questionnaire was tested on online through Qualtrics Survey Software with 69 teacher participants in Canada and Singapore where teacher design work has been emphasized. Of the 69 students who completed the survey, 27 (39.1%) were teachers who have 5–10 years of teaching experiences (Table 6), 25 (36.2%) were teachers who had more than 10 years of teaching experience, and 17 (24.6%) were teachers who have less than 5 years of teaching experience. Half of the teachers (37) had a master’s degree, and 37 (53.6%) of teachers taught in K-12 settings. After applying the survey form, the Kolmogrov Smirnow test was applied to a group of 66 participants after data cleaning to investigate whether the scores obtained via the survey had a normal distribution or not. The distribution of residuals satisfied the assumption (Kolmogorov–Smirnov D = 0.102, p = 0.085).

Table 2 Survey items for data practice
Full size table
Table 3 Survey items for design practice
Full size table
Table 4 Survey items for knowledge creation practice
Full size table
Table 5 Professional teaching practice survey items
Full size table
Table 6 Principal component analysis results for TDK competencies survey (N = 66)
Full size table

Findings

A construct validity was carried out for the structure of the TDK survey form using the exploratory factor analysis to examine a certain number of interrelated variables (Fields, 2013) and identify the relationships between variables explained by conceptually-meaningful factors. The principal components analysis was applied as an exploratory factor analysis and a factor extraction technique. In order to reset the correlations between factors and to help interpret the factors, Varimax vertical axis rotation was used. The number of factors, the lower limit of the item eigenvalue was taken as 1.00 to determine the number of factors. Moreover, the factor load lower limit of each item was taken as 0.40 (DeVellis, 2012; Netemeyer et al., 2003). The KMO sample competency was measured in order to test the validity of the size of the sample statistically, and the KMO value was calculated as 0.678 which is interpreted as normal. Since Bartlett’s Test of Sphericity test were examined (chi- square = 2186.074; df = 903; p < 0.000), it is seen that the data are appropriate for the factor analysis. In other words, the sample size and the correlation matrix is said to be appropriate for factor analysis (Tabachnick & Fidell, 2013).

In the present study, items were removed from the survey during the factor analysis applied in since those items failed to meet the requirement for the value of 0.40. As a result, it was found out that the revised survey form including 43 items had a three-factor structure, that the total variance explained was 48.148%, and that the factor load values ranged between 0.440 and 0.812.

The Cronbach’s alpha coefficient for the whole scale was found to be 0.935, whereas the values of Cronbach’s alpha coefficient for individual factors of the scale ranged between 0.945 (factor 1, 20 items), 0.862 (factor 2, 13 items), and 0.858 (factor 3, 10 items).

Discussion on findings and implications

Despite the limitations of the small sample size, the findings revealed that the TDK competencies survey was a valid and reliable instrument for measuring TDK competencies. The validity and reliability studies of the scale were carried out with 66 teachers after data cleaning. After the EFA, the TDK competencies survey included 43 items from 77 items and had three factors. These factors were Data Literacies Practice (20 items), Design Practice (13 items), and Distributed Epistemic Practice (10 items) in terms of the 3Ds (see Table 7).

Table 7 Revised teacher design knowledge (TDK) competency taxonomy
Full size table
Table 8 Survey themes and items in data literacies practice
Full size table
Table 9 Survey themes and items in design practice
Full size table
Table 10 Survey themes and items in distributed epistemic practice
Full size table

Factor 1: data literacies practice by integrating instruction and assessment

The data literacies practice factor consists of 20 items as one of the most important factors of the TDK competencies survey (Table 8). This factor refers to teachers’ competencies in becoming data-literate teachers who document (ethically access), interpret, visualize, and act on multimodal learning data to empower students.

Echoing Mandinach and Gummer’s (2016) definition of data literacy for teachers, Factor 1 suggests introducing data practice in teacher learning to enhance teacher design knowledge in the transformation of data into instructional decisions (Table 9). Compared to traditional assessments, innovative assessments integrate “assessment and instruction into a seamless, unified activity at promoting learner development through appropriate forms of mediation that are sensitive to the individual’s (or in some cases a group’s) current abilities” (Lantolf & Poehner, 2004, p. 50). In other words, the boundaries between instruction and assessment are blurred as assessment as data-driven decision making in data practice allows teachers to use data effectively to inform their instruction which will in turn encourage students to generate their own multimodal learning data (Table 10). In particular, the data literacies factor aligns with Cope and Kalantzis’ (2016b) claim that data practice mediated by digital technology empowers both teachers and students to ethically access, create, interpret, visualize and act on on-going, real-time big data “as the product of which is a collectively created artifact or solution that cannot be ascribed to individual cognition” (p. 14) to make decisions and evaluate various forms of social interactions.

Data literacies practice as the first TDK competency domain refers to teachers’ meaning-making practice. The process involves: Documenting, Noticing/Interpreting, Visualizing, and Acting on students’ multimodal meaning-making as data. First, teachers need to observe and listen to student learning (e.g., social, emotional, cognitive, metacognitive aspects) through multimodality (e.g., drawing, reading, writing, speaking, art, music, movement) using a variety of documentation formats with the use of technology. Second, this documentation can help teachers notice and interpret a multiplicity of students’ interests, ideas, and authentic problems in students’ multimodal meaning-making by designing playful and collaborative activities. Teachers’ meaningful noticing can give teachers experience in engaging in a variety of learning communities to generate a big idea by consolidating, maintaining, and improving students’ interests and ideas.

Third, teachers should be able to visualize not only the student-generated learning data (e.g., questions, emerging big ideas, improvable ideas) but also learning processes, products, contexts, and interactions with the environment and materials – resulting in communicating shared language and understanding with groups/individual students and other stakeholders. Finally, teachers are also engaged in acting on students’ multimodal meaning-making to increase student agency and advance community knowledge. The important point is that they also need to become data-informed citizen scientists in teaching activities including assessment through the conceptualization of contexts, products, process, and interactions with the environment and materials with the use of technology.

Factor 2: design practice through teaching-by-designing

The design practice factor consists of 13 items on the TDK competencies survey. As leading scholars in educational data sciences, Cope and Kalantzis (2016a) viewed data as evidence of (social) “cognition in the documented social provenance of information” as “the collaborative construction of knowledge artifacts and the quality of reasoning behind a conclusion” instead of assessing “individual cognition in the form of memory and correct application of theorems” (p. 8). They indicated the similarity between the data used for pedagogy and the data used by researchers in the educational data sciences (p. 9) to integrate instruction and assessment in the era of big data for co-designing innovative learning environments with students, researchers, and practitioners. In other words, in authentic learning settings, teachers need to conceptualize contextually-sensitive design principles through continuous reflective inquiry of designing, enacting, analyzing, and redesigning innovative learning environments as described above.

As knowledge workers and designers through knowledge creation, teachers appropriate known design principles through learning, apply known and hypothetical design principles to address complex problems in real contexts in collaboration with practitioners and researchers, test and define (improvise) new design principles to design innovative learning environments (in new designs), and (d) redefine design principles in multiple iterations. These TDK competencies suggest teaching as knowledge creation through teaching-by-designing rather than delivering pre-determined contents, so schools are expected to support and empower teachers to develop capabilities and disposition in advancing existing knowledge (e.g., design principles).

Thus, in the TDK survey, design practice can be defined as teachers’ meta-data literacies practice that allows teachers and students to reconstruct and (re)design available designs for meaning for purposes of data literacies practice. There are the four activities that comprise the design practice process. First, teachers need to engage in appropriating and adapting available design principles that can be reworked as available designs to implement a technology-enhanced curriculum and support students’ multimodal meaning-making in data literacies practice. Second, through teacher-researcher partnerships, it is also essential to build teacher capacity to apply available design principles guided and designed by researchers and other practitioners. Teachers should be able to redefine and apply at least some of available design principles to design and implement a technology-enhanced curriculum where data literacies practice is fostered with the help of technology. Third, it is required that teachers engage in design practice independently, and eventually improvise new design principles through knowledge networks to advance pedagogical understanding and curriculum making. Finally, teachers can redefine design principles in multiple iterations in diverse contexts by ongoing interaction and knowledge co-construction between teachers and researchers/practitioners. This design practice also requires a certain type of visualization of the redesigned design principles as traces of meaning-making that transform teachers into leaders who facilitate further development of more promising design principles and encourage other teachers to better engage in data literacies practice.

Factor 3: distributed epistemic practice for knowledge creation

The distributed epistemic practice factor consists of 10 items on the TDK competencies survey. This factor highlights teachers’ competencies in integrating curriculum elements in terms of Schwab’s (1978) commonplaces (e.g., teacher, student, subject matter, milieu) and design thinking to enrich the collaborative process with students in terms of designerly ways of knowing (Cross, 1982). To continuously design and define innovative learning environments mediated by digital technology, teachers should be engaged in distributed epistemic practice that advances knowledge progressively as teachers empathize with students, identify problems, come up with possible solutions, create a prototype, and test it. As teachers engage in a community of learners as knowledge creators, they should assume agency in using and creating epistemic artifacts (e.g., design principles) to improve their ideas while taking up responsibility to advance the community’s knowledge in addition to individual teacher learning, acknowledging the rights of every student to contribute ideas, embracing diversity in views as strength of the community, and leveraging distributed expertise in a collaborative inquiry leading to the collective advancement of knowledge for all educational stakeholders.

Therefore, distributed epistemic practice in the TDK survey involves teachers’ meta-design practice by engaging teachers in the practice of leadership to support not only students’ multimodal meaning-making in data literacies practice but also design practice by demonstrating adaptability expertise, collaborating with stakeholders, and applying design thinking to co-creation of knowledge. There are the four activities in the distributed epistemic practice process. First, teachers should be able to respond to curriculum and policy changes and adapt available design principles to document the diverse and changing student learning through multimodality using a variety of documentation formats by seeking out, appropriating, and adapting available design principles. These available design principles can be reworked as available designs to implement a technology-enhanced curriculum to support data literacies practice and design practice. This adaptability is important to cope with unexpected and emerging needs of student learning in the implementation of a technology-enhanced curriculum. Second, teachers can participate in collaborative inquiries into their own teaching practices and continually integrate new knowledge into data literacies practice and design practice. These collaborative engagement activities are important to notice a multiplicity of students’ interests, ideas, and authentic problems and advance the community’s knowledge by applying available design principles.

Third, it is required for teachers to incorporate design thinking into emerging changes in teaching, curriculum, or policy changes to communicate learning processes, products, contexts, and interactions with the environment and materials with groups/individual students and other stakeholders by improvising new design principles through knowledge networks. Finally, teachers need to engage in the practice of leadership to support students’ multimodal meaning-making by visualizing the redesigned design principles in multiple iterations in diverse contexts to advance data literacies practice and design practice.

Conclusion

The current work developed the survey instrument to assess teacher design knowledge (TDK) competencies drawing upon a previous publication on a competency taxonomy for TDK (Kim, 2019) in technology-enhanced learning environments. This TDK survey included 43 items categorized into three factors including data literacies practice, design practice, and distributed epistemic practice. The first TDK competency domain, data literacies practice refers to teachers’ meaning-making practice by documenting, noticing/ interpreting, visualizing, and acting on students’ multimodal meaning-making as data through multimodality using a variety of documentation formats with the use of technology. Design practice, the second TDK competency domain, is defined as teachers’ meta-data literacies practice that allows teachers and students to reconstruct and (re)design available designs for meaning for purposes of data literacies practice. This design practice engages teachers in appropriating available design principles for students’ multimodal meaning-making in data literacies practice, applying them through teacher-researcher partnerships, improvising new design principles through knowledge networks to advance pedagogical understanding and curriculum making, and redefining these design principles in multiple iterations through expansive learning. The third TDK competency domain, distributed epistemic practice, refers to teachers’ meta-design practice that engages teachers in the practice of leadership to support not only students’ multimodal meaning-making in data literacies practice but also design practice by demonstrating adaptability expertise, collaborating with stakeholders, and applying design thinking to co-creation of knowledge. To further develop and validate the instrument, the author is also in the process of conducting systematic but flexible design-based research with teachers who engage in the work of design by extending their pedagogical repertories to facilitate both collective and personal knowledge construction.