Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Introduction

Finnish teachers are highly educated professionals. Whereas the traditional lecturing role of a teacher is still seen as essential, there are also many other roles, such as guidance and collaboration with other professionals, that are coming to be seen as more important parts of the teachers’ profession in Finland (Krokfors et al. 2010). However, as noted in the ITL (Innovative Teaching and Learning) study (Norrena 2013), even though twenty-first century skills are recognised and mentioned in the curricula of Finnish comprehensive schools, the schools and, especially, individual teachers vary greatly in their ability to facilitate the development of twenty-first century skills. The teachers consider teaching twenty-first century skills to be difficult (e.g. Niemi 2012) and it is generally up to individual teachers’ discretion whether they include elements of innovative teaching and learning in their instruction. Therefore, teacher education units have a central role in contributing to this pedagogical evolution. Teacher education focused on developing twenty-first century skills has the potential, with a research-based curriculum and carefully designed learning practices, to provide new teachers entering schools with a better foundation to meet the many challenges of twenty-first century learning environments (see Kong et al. 2013).

The Finnish national interest in fostering twenty-first century skills has highlighted a need for – and interest in the development of – tools and methods for teaching and assessing such skills. Assessment in Finland is often formative, based on constant evaluation of an individual student’s development in different subjects. In addition, the changes in the Finnish school curriculum in August 2016 towards phenomenon-based instruction with an emphasis on more interdisciplinary and generic skills and competencies have created a need for new forms of assessment. In this regard, technology-enhanced systems that enable formative assessment of complex performances involving collaboration are becoming more essential (Binkley et al. 2012; Van Aalst 2013).

Finland’s participation in the ATC21S project was a step towards better understanding of the assessment of more complex skills. While today’s international and national standards primarily measure core subject performance (in math, science and reading), ATC21S designed new assessment prototypes to help education systems include the twenty-first century skills that are essential to performing better in those core subjects. Finding technical solutions to meet schools’ everyday pedagogical goals is an interesting and ongoing challenge. The work done in the ATC21S project continued as part of the “Preparing teacher students for 21st century learning practices”, PREP21 project (Häkkinen et al. 2017; PREP21 2015; Pöysä-Tarhonen et al. 2016). The purposes of this study are, firstly, to acquire better understanding about the Finnish early stage pre-service teachers’ dispositions towards teamwork and collaboration and, secondly, to assess their existing level of collaborative problem solving skills by using the novel technology-enhanced assessment system of the ATC21S portal.

By linking the students’ dispositions and self-assessments to objectively measured levels on learning progressions of collaborative problem solving, the aim is to investigate existing connections and disconnections between these measures and answer the following research questions:

  1. 1.

    What is the current level of collaborative problem solving skill among the two selected groups of pre-service teachers?

  2. 2.

    What kind of relation exists between teacher education students’ collaboration and teamwork dispositions and assessed collaborative problem solving skills?

Method

Participants and Context of Study

The participants of the study were second-year teacher education students (n = 24, 21 female, 3 male) from one Finnish University. The teacher training program of this university follows phenomenon- and inquiry-based learning approaches. The phenomenon-based curriculum integrates, for example, the study of educational science and research methods into inquiry-based study projects. In addition to the phenomenon-based approach, all the students study in home groups. Different home groups have different profiles in terms of their study contents and methods. The students of this study were following two divergent teacher education programs in their home groups. Common for both of these programs was that they apply phenomenon-based, collaborative modes of studying and their students are, hence, supposed to be experienced in engaging in productive collaborative activities, including collaborative problem-solving activities. Study projects with schools are also included in both of these study programs. Active agency for own learning is emphasized in these study programs in terms of both students’ own studying and in promoting pupils’ learning at school.

Group A consisted of 12 students from a study program specializing in technology-enhanced learning (TEL). The goal of this group was to envision and experiment with the use of learning technologies with students in school settings. Hence, these students use also in their own studies multiple tools and technologies (e.g. personal mobile devices/tablets, social media, games) for individual access, manipulation and analysis of information as well as for communication, sharing and joint knowledge construction with peers. In comparison to group B, group A utilized the phenomenon-based approach more thoroughly and participated only minimally in traditional lectures. Another dimension that was more present in group A was collaborative ‘teachership.’ The aim in this program is to model collaborative teaching for students by coaching and supervising them as a team of teacher educators. In distinguishing the two groups, we called group A the “Technology” group.

Group B consisted of 12 students following a program focusing on STEM-related themes, especially in science and mathematics. This group relied on the inquiry-based curriculum but also participated in lectures more than group A. Although this group had an emphasis on communities of teachers, they were not given a model of collaborative teachership in their own studies. They had only one teacher at a time guiding their studies. We called this group the “Inquiry” group.

Measures

To assess the pre-service teachers’ CPS skills from different perspectives, two measures were combined. First, a PREP21 self-report questionnaire was used. A set of questions based on the work of Wang et al. (2009), also applied as part of the PISA 2015 background questionnaire, was created to measure dispositions towards cooperation, negotiation and guidance. In this approach, these student dispositions are defined as general attitudes towards collaboration, collaborative problem solving and teamwork. Dispositions refer, thus, to students’ broader attitudes, beyond any particular collaborative learning situations or contexts. Accordingly, these dispositions are supposed to predict students’ performance in collaborative problem-solving activities (OECD 2017). Also, obtaining a better understanding of teamwork as a set of skills and dispositions provides the grounds for deeper exploration of how students may acquire these skills and how instruction could be better designed to assist students in developing and applying these skills in professional settings (Hughes and Jones 2011).

The items referring to students’ dispositions towards collaboration, collaborative problem solving and teamwork were scored on a 7-point Likert-type scale, from 1 (not at all true of me) to 7 (very true of me). The subscales were based on responses to a PREP21 survey of a larger sample (N = 263) of Finnish pre-service teachers. For the internal consistency of measured subscales, Cronbach alpha was used. The reliabilities were measured as (α = 0.74) for the Cooperation subscale and (α = 0.75) for the Guidance and Negotiation subscales. These can be considered adequate reliabilities of scale (Nunnally 1978). First, negotiation is seen as a central element of teamwork, because an individual needs to negotiate and adjust his/her actions according to the surrounding group. The Negotiation subscale comprises variables related with the ability to listen to others, flexibility and openness to others’ thoughts and ideas. Negotiation was measured with six items: “I am a good listener”; “I enjoy seeing my classmates be successful”; “I take into account what others are interested in”; “I am flexible when working with a team”; “I enjoy considering different perspectives”; “I am open to all sorts of opinions”. The subscale of Guidance includes the teacher education students’ dispositions towards their skills to guide and mentor their other team members. The disposition of guidance was measured with six items: “I like to be in charge of groups or projects”; “I enjoy sharing ideas”; “I convince others to see things my way”; “I enjoy exchanging ideas”; “I like convincing peers”; “I enjoy bringing a team together”. The subscale of Cooperation includes teacher education students’ dispositions towards working together as a team. Cooperation was measured with four items: “I prefer working as part of a team to working alone”; “I find that teams make better decisions than individuals”; “I find that teamwork raises my own efficiency”; “I enjoy cooperating with peers”.

In addition to the survey of dispositions towards collaboration and teamwork, an assessment portal, ATC21S, was used to assess the students’ skills over the course of CPS activities. Each pair of students completed one bundle of five assessment tasks over a period of 90 min. Group A completed Laughing Clowns, Plant growth, Balance, Olive oil and Game of 20. For Group B, Game of 20 was replaced by Small pyramids. These tasks have been described by Care et al. (2015). They are complex game-like tasks, mainly in the science and math domains, related both to curriculum content and to generic skills. The participating pairs proceeded well in the assessment. All of them could either enter or finish the last task (Game of 20 or Small pyramids). Moreover, in the ATC21S portal, students’ completion of the assessment tasks yielded log file data. The data generated were captured in a process stream data file, and patterns in these data were automatically coded as indicators of the CPS elements (Adams et al. 2015; Hesse et al. 2015). Furthermore, the tasks captured social and cognitive components of students’ CPS skills. Each of the skills could thus be scaled based on the actions taken by the students, which were collected as process data, together with the online chat discussions that took place while performing CPS tasks.

The scoring itself took into consideration students’ actions as they moved through the tasks. The process data consisted of distinct keystrokes and mouse events that indicated exploration of the task environment, such as typing, clicking, dragging, cursor movements, hovering time and action sequences, all of which explicitly demonstrated students’ thinking processes and skill levels. The log file data from the assessment tasks were processed by the Assessment Research Centre of the University of Melbourne. All of the user actions and chat messages were recorded and time-stamped. The files generated for the automatic records of student-task interactions are referred to as session log files (Adams et al. 2015). Next, MySQL database architecture was used to record the interactions within the task environment. The scoring engine then automatically coded and scored data to produce reports for teacher and student use. Figure 7.1 provides an example of the reports on social and cognitive skills retrieved from the portal. This particular student (fifprr0002a) was estimated at a Level 5 for her cognitive skills and a Level 6 for her social skills.

Fig. 7.1
figure 1

Learning readiness reports from ATC21S™ portal: cognitive and social skills

Full size image

Data Analysis

Scores on respondents’ skill level estimates were firstly drawn from the ATC21S assessment portal by utilising the Rasch modelling on the ConQuest™program (Wu 2007), a multi-aspect test software. All analysis was performed with ConQuest™, using a Partial Credit model with Guass-Hermite Quadrature estimation with 15 nodes. These skill level estimates were then used to create the skill level reports. In accordance with the procedures of Rasch modelling, the average of the task scores was set to zero and the difficulty of the item was presented as an estimate describing the level of the students based on their results on the bundle of tasks they had completed. Each student received a Weighted Likelihood Estimate score (WLE), which could vary from −4 to 4 on both social and cognitive dimensions. The report displays are based on the Weighted Likelihood Estimate scores (WLE) distributed on different levels of learning progression, presented in Table 7.1.

Table 7.1 Range of WLE scores in the ATC21S portal corresponding to the learning readiness levels
Full size table

The participants were analysed as two separate groups based on their study group. SPSS v 22 was then used to investigate confidence intervals (using T-tests), descriptive statistics and correlation in the whole set of data and between the two groups. The statistical significance of mean differences was tested using one way analysis of variance. Due to the small sample size, the correlations were calculated with non-parametric Spearman’s rho. We used general criteria to interpret the correlation coefficients: 0.10–0.29 for weak, 0.3–0.49 for moderate, 0.5–0.69 for strong, and above 0.7 for very strong associations between variables (c.f., Cohen 1988).

Results

The descriptive statistics for the scores received from the ATC21S assessments and PREP21 questionnaire dispositions for all participants are presented in Table 7.2. The ATC21S WLE scores are presented separately for social and cognitive skills. Pre-service teacher students’ social skills were reported with a mean of 1.92 (SD = 0.65), which falls in the highest level of learning progression: Level 6. Their cognitive skills were reported with a mean of 1.62 (SD = 1.02), which falls in Level 4 of learning progression. The level of social skills was fairly consistent at the very top level, but cognitive skill levels varied and were overall at lower level compared to social skills. The dispositions from the PREP21 questionnaire were at a rather high level. The mean of Guidance was 5.32 (SD = 0.65), the mean of Cooperation was 5.25 (SD = 1.24) and the mean of Negotiation was the highest, at 6.08 (SD = 0.75) on a scale of 1 to 7. Due to the small sample size, and the fact that the sample responses were not normally distributed, correlations were calculated with non-parametric Spearman’s rho. Negotiation correlated significantly and positively (r = 0.57) with the ATC21S social skills WLE score. ATC21S cognitive skills WLE score did not have statistically significant correlations on any measured variables. The disposition variables also correlated significantly with each other. Negotiation correlated strongly (r = 0.57) with cooperation and moderately (r = 0.41) with guidance. Also cooperation correlated moderately (r = 0.41) with guidance.

Table 7.2 Descriptive statistics, 95 % confidence interval of difference, Spearman’s rho
Full size table

Due to the small number of respondents, it was also possible to examine each student’s scores and dispositions individually, based on their study groups. Table 7.3 presents these individual measures separately. The WLE scores were indicated as levels of learning readiness based on Rasch modeling of the item difficulties. There were no significant differences in the mean scores between the two groups of pre-service teachers. Still, group A “Technology” scores were more consistent, with lower standard deviations on both social and cognitive WLEs when compared to group B “Inquiry”. Group A students also generally indicated slightly higher dispositions than group B students. But only on negotiation was the difference statistically significant (p < 0.05), where the group A mean was 6.40 and the group B mean was 5.76. When Social skills WLE are associated with negotiation dispositions, it can be interpreted that students from both groups utilized their negotiation capacity well in the social processes of collaborative problem solving. When examining the dispositions on individual level, it is possible to recognise that four students from group B indicated their negotiation dispositions below 6 and two below 5. In group A, only one student’s dispositions were below 6. When examining the results of individual student ID 5a, it can be seen that her social skills WLE were measured on the lowest level (1.29) in group A. When examining the results of individual students from group B it can also be seen that student ID 21a had reported her dispositions as rather low on cooperation (3.00) and negotiation (3.33), which are actually the lowest ratings of all. Her skills were also measured as the lowest with ATC21S. In this particular case the ATC21S measurement and student’s own dispositions met exceptionally well. Still, this particular students’ pair ID 21b had very high ratings on the portal, where both her social and cognitive skills were measured at the very top level. Despite her high measured skills, her dispositions did not indicate high expectations regarding her skills on teamwork and collaboration when compared to other high scoring students. The only clear difference, when compared to her pair 21a, was that her dispositions on negotiation were clearly higher (4.67). This indicates that negotiation dispositions were clearly associated with this student’s measured social skills.

Table 7.3 Individual portal scores and dispositions
Full size table

Discussion and Conclusions

Recent developments in technology-enhanced assessments have made it possible to evaluate complex performance such as collaborative problem solving more effectively. In this study, we used the ATC21S to assess teacher education students’ collaborative problem-solving skills and a PREP21 self-report instrument to measure more general collaboration and teamwork dispositions. The results indicate that the current level of collaborative problem-solving skills among these students is generally high; measured levels of social skills were especially high, as compared to cognitive skills. Social skills were also connected positively with collaboration and teamwork dispositions – in particular with negotiation. However, the cognitive skills scores did not correlate with teamwork and collaboration dispositions. This indicates that the social aspect of collaborative problem solving is probably the key for success in these kinds of shared tasks.

The respondents were representing two different study groups with slightly different implementation of their study programs. Group A “Technology” focused particularly on technology-enhanced learning, and group B “Inquiry” followed the STEM-related program. Common to both of these programs is that they apply collaborative modes of studying. It can be assumed that these students have been trained to be more familiar with productive forms of collaboration and collaborative problem solving than an average group of students. As compared to group B, group A had a slightly stronger focus on phenomenon-based curriculum, with hardly any lectures in their studies. Furthermore, they also received a model of collaborative teaching as they were coached and supervised by a team of teacher educators.

Group A “Technology” had consistently higher results on social skills when compared to group B “Inquiry”. Both groups had higher social skills than cognitive skills, but there were no significant mean differences between the groups. The finding that negotiation had statistically significant and positive correlation with social skills WLE scores measured by ATC21S supports the assumption that there is an empirical connection between these two independent measures.

Given the adaptations that a society based on knowledge and competence demands from school pedagogy, it must be remembered that teacher education needs to be adjusted to meet the challenges. Pre-service teachers have a central role in developing twenty-first century learning practices and promoting skills such as collaborative problem solving in future schools. In Finland, autonomy is typical for the teaching profession, which also means that teachers often work too independently, sometimes alone. As the skills to solve complex, cross-curricular problems in teams become more important in our society, teachers should acquire these skills also by themselves. In general, the adoption of new pedagogical innovations has been unsuccessful, primarily because too little attention has been paid to teacher’s’ own learning processes (Lieberman and Pointer Mace 2008). Thus, we argue that the task of teacher education is to guide these processes.

Pre-service teachers are themselves the result of traditional school culture, which strongly influences their assumptions regarding good teaching models (i.e., favouring models featuring a traditional teacher-led approach) (Mäkitalo-Siegl et al. 2011; Schratzenstaller 2010; Webb and Mastergeorge 2003). We believe that pre-service teacher education could be a powerful means of sparking long-term change in the field. To create change in schooling, pre-service teachers first need to learn how to adapt to the new learning culture. One of the specific aims of the PREP21 project is to outline the analysis and pedagogical designs regarding students’ collaborative problem solving skills and the related pedagogical practices in pre-service teacher training programs. Based on this experimental study it can be inferred that skills needed in successful collaborative problem solving measured by ATC21S benefit from collaborative practices of instructional methods in teacher training. Using this web-based portal to measure collaborative problem solving in the pre-service teacher education context was the first pilot in advancing the assessment of students’ complex skills. It can be concluded that these tasks are welcome and well suited to pre-service teacher training.

In research to follow, we will apply ATC21S assessments in the context of teacher education on a wider scale, in which the assessment session is followed by debriefing of students’ scores. With larger numbers of respondents, we will examine the interesting associations between teamwork dispositions, self-assessment and ATC21S. In addition, by monitoring the performance of students during the tasks by applying online measures of their performance (e.g. by capturing screen activity) and combining it with subjective data (e.g. cued retrospective interviews) (see Pöysä-Tarhonen et al. 2016) we might be able to better understand the individual differences monitored over the course of this study.