Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Context

Singapore is a small nation-state in South-east Asia, with about five million residents packed into all of 710 km2 of land. It is home to a multi-ethnic and multi-lingual society of Chinese, Malays, Indians and people of many other ethnicities. Education is an important pillar of the Singapore social and economic architecture. The 2014 education budget of more than 11 billion Singapore dollars (about USD 9 billion) represents about 20 % of total government expenditure (Ministry of Finance, Singapore 2014) and is the second highest amount after expenditure on defence. This reflects the national priority given to education.

Formal schooling starts at age six in Primary 1 (equivalent to grade 1). Virtually all the half million students are enrolled in 357 publicly funded primary schools, secondary schools and pre-university institutions. Singapore has a national curriculum that provides equitable access to a broad and holistic education that includes the study of English, Mother Tongue Languages (such as Mandarin, Malay and Tamil), mathematics, the sciences, physical education, humanities and the arts. These subjects are complemented by co-curricular activities and community service programmes that develop life skills and socio-emotional competencies. All public schools base their teaching and learning programmes on the national curriculum and subject syllabuses. The subject syllabuses are reviewed regularly to ensure that they remain relevant for the future.

In 2009, Singapore developed a 21st century competency (21CC) framework to guide the development of its national curricula. The framework (Fig. 9.1) articulates the competencies that would enable students to grow into confident and concerned citizens with the necessary attributes and skills to learn continuously, work effectively in teams, exercise initiative, take risks and strive for excellence (Ministry of Education, Singapore [MOE] 2010). The framework identifies three sets of enabling 21st century skills:

Fig. 9.1
figure 1

Framework for 21st century competencies and student outcomes © Ministry of Education, Singapore (Reproduced with permission from the Singapore Ministry of Education)

Full size image
  1. 1.

    Civic literacy, global awareness and cross-cultural skills;

  2. 2.

    Critical and inventive thinking; and

  3. 3.

    Communication, collaboration and information skills.

The 21CC framework extends the work of Thinking Schools, Learning Nation, an education reform movement that began in 1997 to nurture a more thinking and inquiring mindset among Singapore students (Ng 2004; Sharpe and Gopinathan 2002). With the implementation of the 21CC framework, efforts were mounted to expand opportunities for all students to develop these competencies. For example, elementary schools enhanced their approaches to the teaching and learning of art and music to better develop creative capacities and personal, cultural and social identity in students (MOE 2011).

ATC21S Development in Singapore

It was during this period of heightened interest in the teaching, learning and assessing of 21CC that Singapore joined Australia, Finland, the United States, the University of Melbourne, and three international companies – Cisco, Intel and Microsoft – in founding the Assessment and Teaching of 21st Century Skills (ATC21S) project. Essentially, the ATC21S project sought answers to the following key questions (Griffin et al. 2012):

  • What are the 21st century competencies?

  • How can teachers teach them? How do students acquire them?

  • How can students demonstrate them in measureable ways?

The ATC21S project in Singapore was spearheaded and funded by the Ministry of Education and drew in collaborators from the National Institute of Education, Nanyang Technological University. Twelve schools were also recruited as our partners in the project. Participating in the ATC21S project enabled Singapore to work with the international research community to establish assessment practices for 21CC, and more importantly, to find ways to automate such practices. These serve to complement and enhance our national efforts in the teaching, learning and assessment of 21CC. As the project developed intensively in the ensuing months, Singapore was heavily involved in the four critical phases of task concept check, cognitive laboratories, pilot studies and trials in the development of the task prototypes for the two sets of 21st century competencies – learning through digital networks – information and communication technology (LDN-ICT) and collaborative problem solving (CPS).

Prior to the actual data collection for the ATC21S project, it was hypothesised that the teachers and students involved in the study would have had limited exposure to the types of assessment tasks that were being developed in the project. Therefore, beyond the data that were specific to the ATC21S research questions, the Singapore team extended the scope to capture the teachers’ and students’ experience in their engagement with the tasks. The additional questions included: Did the teachers think that the tasks were assessing 21st century competencies? What aspects of the assessment and tasks did the teachers think would engage students, given the socio-cultural context of Singapore classrooms? What troubled or encouraged teachers when they thought about using the tasks for teaching, learning and assessment? Similarly, what aspects of the assessment and tasks did their students find engaging? And what aspects did students think were challenging?

In this chapter, we share the main findings and reflections from our inquiry into these questions within the context of the iterative process being used by the ATC21S tasks to further refine both the tasks and assessment practices. It is hoped that by sharing the voices of our teachers and students through the task development process, the international community can gain useful pointers in the collective efforts to assess 21CC attributes.

Method

In all, 87 teachers/education officers and a sample of about 2,000 students aged 11, 13 and 15 from four elementary and eight secondary schools in Singapore were involved in the four phases of task concept check, cognitive laboratories, pilot studies and trials (see Table 9.1). Griffin and Care (2015; Chap. 1) describes the processes involved in the four phases.

Table 9.1 Number of ATC21S participants from Singapore
Full size table

The 12 schools accepting the invitation to participate in this project had ensured that all teachers and students taking part had given their consent with the understanding that they could withdraw at any point if they wished. Table 9.2 summarises the sources of data for the current chapter.

Table 9.2 Sources of data for this chapter
Full size table

Responses to the ATC21S Tasks

In our interviews of teachers and students, we asked for their views about using the ATC21S tasks for teaching, learning and assessment of 21CC. We also asked them to tell us which aspects of the tasks engaged, encouraged or challenged them. In this chapter, we have organised our teacher and student responses to the ATC21S tasks around six areas:

  1. 1.

    Relevance to the teaching, learning and assessment of 21CC

  2. 2.

    Engagement with tasks

  3. 3.

    Seeking meaning in content-rich and content-free tasks

  4. 4.

    Collaboration

  5. 5.

    Introducing ambiguity in tasks

  6. 6.

    Tools and technical issues

Relevance to the Teaching, Learning and Assessment of 21st Century Competencies

In general, the teachers saw the potential of using the ATC21S tasks for the teaching and learning of 21CC, barring customisation to suit local contexts. In reviewing the learning through digital networks tasks, teachers felt that the tasks were, in principle, aligned to the targeted constructs of functioning as consumers and producers of knowledge; and developing and sustaining social and intellectual capital. One elementary school teacher described how these tasks could help build the skills of “communication, reaching consensus when different ideas are there” which were “important, essential skills” that could support his school’s 21CC program on “inventiveness”. The CPS tasks were good exemplars of how to build in the need for students “to work together…and to really see strengths in the other person”.

In previewing the tasks at the concept review stage, our teachers anticipated that the ICT-delivered tasks would likely appeal to their students whom they regard as “IT-savvy”. Indeed our students, who were more exposed to pen-and-paper tests, were generally intrigued by the novelty of the tasks. However, 21st century competencies are not about the use of ICT alone. Instead, they are about the potential that judicial use of ICT can have on the transformation of key aspects of assessment, particularly the measurement of 21st century skills, such as metacognition, creativity and collaborative problem solving (Binkley et al. 2012), that are difficult to assess through pen-and-paper tests. The ATC21S tasks were good exemplars where there was clear leverage on ICT and social networking tools for online collaboration. However, one teacher observed that the ATC21S tasks have not made good use of the ICT platform to provide timely feedback to the students:

I do not see any formative feedback given to students in the task. Students are not aware of whether they got the answer right or wrong, and how do they improve based on the mistakes they have made in solving the questions?

One student reiterated this observation:

I received no feedback.

The teacher and student were not wrong in their expectations. Experts who worked on the ATC21S white papers have pointed to assessment innovation through advanced Web 2.0 technology that could tailor assessment and feedback to students even while they were working on the tasks (see Wilson et al. 2012). This is indeed an area of work-in-progress for the ATC21S team. In fact, data collected during the trials would provide a rich source of information to develop the learning analytics for the two sets of competencies, a step towards designing an automated system that could provide just-in-time probes to measure learning progress and to provide feedback. As one teacher put it, it would be excellent if students were learning “without realising that they are being taught”. Such a system could also then address teachers’ concern about the need to differentiate the tasks for students at different levels of proficiency in the 21CC.

Engagement with Tasks

Students were engaged by the dynamic and interactive nature of the tasks. For example, in one of the ATC21S tasks, students could “send” weights to each other to balance a beam. One student said: “I enjoyed the interactive tasks”. Other students described the tasks as “fun, attractive, interesting”. In fact, students’ overall engagement level with the tasks was very high and many indicated that they would like to do similar tasks in school. From the post-task survey we conducted during the pilot and trials, close to 9 in 10 students agreed that the ATC21S tasks were interesting, more than 8 in 10 enjoyed solving the tasks together with their partner and more than 7 in 10 students preferred this mode of assessment to traditional pen-and-paper tests. These findings are consistent across all age groups.

Although students described many of the tasks as “fun”, it did not mean that they found the tasks intellectually unchallenging. In fact, most of our students did not find the tasks easy, as one student summarised it:

The task sets you thinking – puts thinking and analytical skills to use.

The appropriate level of intellectual challenge in a task plays a role in engaging students – studies have shown that students may become disengaged when tasks are not challenging enough (e.g., Hayes 2008). Conversely, task designers also need to guard against tasks that are perceived to be too challenging (Brophy 1987). Pitching the task appropriately for students is therefore an important consideration in engaging students in learning. It is not unreasonable to believe that this principle does apply to the learning of 21CC. In fact, the ICT platform that delivers the task can potentially be leveraged to differentiate task difficulty for different students.

Seeking Meaning in Content-Rich and Content-Free Tasks

How can 21st century competencies be best learned and assessed? Should they be embedded in content-rich tasks or in tasks that require very little disciplinary knowledge? This was one of the questions that the ATC21S team wanted to find some answers to. It therefore commissioned experts to develop both types of tasks. Different prototypes that drew on varying degrees of content knowledge were developed as contexts for measuring collaborative problem solving, ranging from content-rich tasks based on specific scientific or mathematical concepts (for example, Game of 20), to relatively “content-free” tasks that required students to recognise general patterns or rules from a real-life scenario (for example, Hot Chocolate).

Content-rich tasks generally sit well within most curricular frameworks that are designed along disciplinary lines, making it easier to identify the teachers who would teach the 21CC within disciplinary content. On the other hand, there are also concerns that the content within a task might alter proficiency estimates on the 21CC construct (Wilson et al. 2012).

Based solely on qualitative responses, we found that both teachers and students tend to struggle to find “purpose and meaning” when encountering content-free tasks. A student during a cognitive laboratory session thought aloud: “I don’t get what you are supposed to do here. Asking to make a line appear but what is the line for? Purpose?” In another save/print task, a student said: “Don’t understand what this is for… Since it is a tutorial to teach me something, just follow blindly what they are telling me.” Teachers similarly searched for coherence and meaningfulness of the various activities, as illustrated in several teacher voices on the poetry task:

What movie are you creating? What are the objectives? What is the feedback? What is the point of the exercise?

It’s testing the ICT skills, not the poem. They have the skills without learning anything.

The students just do, don’t know why.

Meaning making is important.

There is no link between the literary elements and the task. It’s not really about this poem, what does it capture?

While it could be argued that these exemplified the cultural mind-set of teachers who were discipline-centric, or that these were unfamiliar experiences for teachers and students who tended to focus on assessment of content knowledge, it was also possible to understand from that context that pedagogically sound acquisition of content could actually help build the very competency measured in this task. As explained by a participating teacher:

…there is room to build students’ knowledge of literary elements and devices through the resources already made available in the scenario and exchange of views with partners to build new insights in the understanding of literary works. For example, while Singapore students may not understand the significance of the use of “Jim Crow” and the historical element of the Merry-go-round piece in America, actual exchange of views with other students can help them build this social understanding and capital.

Related to this is the question of whether the quality of thinking is integral to 21st century education. Beyond demonstrating the ability to use ICT and networking tools, is the quality of the output, in terms of what students write and create, and how they reason and justify their answers during the task, integral to 21st century teaching, learning and assessment? In terms of building intellectual capital, teachers expressed the view that the quality of the ideas and knowledge generated from the task was equally important to the 21st century skills that are being measured.

Perhaps the crux of the matter is not so much the question of whether the task is content-rich or content-free. Regardless of whether it is anchored in a discipline or not, a more important issue is whether students could find meaning and authenticity when working on the tasks. The tasks could be scaffolded to build up towards meaningful goals, as meaning making is an important aspect of learning (Perkins 2009). As a 15-year old commented: “I was bothered from the beginning on what was the aim of the activity”. There is therefore room to consider this perspective of meaning making for teachers and students in the design of the tasks.

Collaboration

There was considerable deliberation within the ATC21S community on linking the design of the tasks to the accurate assessment of collaboration. The key debate revolved around the balance in the provision of symmetric and asymmetric access to information that would facilitate meaningful collaboration. Each approach seemed likely to solicit different aspects of collaboration skills and provide avenues for the development of these different dimensions.

The developers for the learning through digital networks tasks believed that the same information should be available to all players in a collaborative setting, while the developers for the collaborative problem solving tasks ensured collaboration by giving different collaborators access to different sets of information. We believe that the ideal scenario would be a good balance between the two across a comprehensive range of tasks. We made some observations on how our students collaborated based on the current tasks. Our students responded to “collaboration” tasks in several ways – (i) one student worked out the solution while the partner passively agreed to the solution; (ii) one student worked out the solution while the partner verified the answer; (iii) the two students discussed the problem and worked out their solutions collaboratively. Our students tended not to collaborate on tasks where they and their partners were presented with the same information. Collaboration was not seen to be critical in accomplishing the task, as students could attempt the questions without working with others. In other words, students saw the problem solution as the larger goal and sidelined the collaboration when they perceived that they could solve the problem independently.

During the interviews following the cognitive laboratory, students explained that the different pace at which their partners completed each task affected the quality of collaboration (each group had between three to four collaborators during the cognitive laboratory for learning through networks). One student reached a collaborative task in Arctic Trek much earlier than her team-mates. She initiated a chat but later decided to skip that collaborative chat and move on to the next task after having no response from her partners, none of whom had arrived at this part of the task yet. She said: “It’s frustrating when my friends don’t answer back”.

One aspect that students liked about the collaboration tasks was the opportunity for negotiation and decision-making among members who collaborate. Our students liked the opportunity of listening to “conflicting opinions”, of “giving my own opinions” and of “seeing friends’ thoughts in Webspiration (the poetry task)”. The students did not often encounter such opportunities in their regular classrooms, as one student commented: “This kind of online negotiation, discussion and decision-making is not common during school work.” Indeed, learning how to negotiate and make decisions in a team are important competencies to develop.

One of the things that we are curious to find out from the ATC21S data is the impact of student ability on collaboration outcomes. For example, we observed that when a very academically able student was paired with an academically weaker student, there appeared to be domineering behaviour by the more able and articulate student to take the lead in the task. We are not sure if this observation was an isolated event or that prior ability of students could have an impact on the levels and quality of collaboration. Further analysis of the actual ATC21S data collected will be helpful in answering this important question.

In one of the interviews, one student told us: “I find it very difficult to answer and work with others through on-line collaboration”. It made us wonder if measures of collaboration would also depend on the modality (e.g., using online chat versus audio chat) and platforms (e.g., offline versus online) of collaboration. This would be an interesting follow-up study that could help improve the design of 21CC collaborative tasks.

Introducing Ambiguity in Tasks

One consistent observation made by our students and teachers was that the ATC21S tasks were less well-defined than the tasks in traditional tests. Students generally had to figure out the problem they had to solve and the approaches they needed to take to solve it. For example, in the Warehouse task, where the objective is to secure a warehouse by correctly positioning security cameras, collaborators are presented with different information – one can see and place cameras while the other sees only yellow beams (which show the coverage of the cameras placed) but not the cameras. Students need to realise that they have control of different parts of the problem, and in this case, figure out what the yellow beams represent. In the cognitive laboratory, we often heard students say:

What is the question?

What am I supposed to do here?

How am I supposed to do this?

The instructions are not clear.

One student said there was “not enough information given in the tasks on how to use the relevant websites to answer the questions”. The younger elementary students described the task instructions as “confusing”. The students were searching for “clear instructions,” as one student put it – something they were more used to in the tests they have usually encountered. Students said they have to “figure out on my own”, “spend a lot of time to figure out what was expected” and “when I found it, which course of action I should take, then much clearer. The activity became much more understandable.”

One teacher made a keen observation about the tasks: “Introduction of ambiguity into the tasks creates space for students to think, inquire and collaborate”. Indeed, many of the ATC21S tasks were deliberately designed to be less structured and more ambiguous than traditional assessments, and to provide collaborators with different ‘views’ of the problem. This created space for students to think about what the problem was and to collaborate and work out different ways to go about solving it. This is an attempt to better reflect the reality of 21st century contexts where problems are ill-defined, information/expertise resides with different sources and solutions are neither immediately obvious nor straightforward (NRC 2011).

While we support the introduction of ambiguity into the 21CC tasks, we suggest that task designers pay more attention to how the goals and instructions of the task are crafted, providing clarity where ambiguity is not intended so as to reduce confusion and not to discourage students from continuing or even getting started with the tasks. For instance, a 15-year old student commented during the cognitive laboratory: “Reflect on the poem. What are they asking in this question? Is it evaluation of poem or usefulness of poem?” The extent of ambiguity introduced should perhaps also take into consideration the age and proficiency level of the students. Younger students or students starting from a lower level of proficiency might require more scaffolding and feedback to help them stay engaged with the tasks instead of giving up because they were too “confused” or even discouraged (see for example Kirschner et al. 2006). Already, some of the ATC21S tasks are good exemplars of problems for which scaffolding has been built into the design. For example, in a task based on a numbers game, students first play against the computer on their own to understand how the game is played, before they move on to work with their partner to play against the computer. In another task, students can review the outcome of the previous page to understand how the puzzle is solved and apply their learning to the problem on the next page (which is built on the previous puzzle). In other words, the problems are designed to be increasingly less well-defined as the students progress through the task. These are important task design considerations, especially when working with different learners.

Tools and Technical Issues

Choice of online tools and support for technical issues are important in task design to minimise confounding factors in measuring student proficiency levels in the 21CC. The design of tools that students need to use in tasks should be user-friendly and intuitive so that students are not hampered by the complexity of the tools in progressing within the task. In the poetry task, students across all age groups had difficulty using the mind mapping tools – many found it confusing and most challenging. One student who skipped the page said during the post-task interview that he knew what a mind map was and how to draw one, but he did not know how to use the tools in the task to do that. Another student thought aloud: “Cannot seem to do anything… This page is confusing, how to use this? Not sure what to do.” The student later explained that the task did not give specific instructions on how to use the tools and the tools were not intuitive. For example, the pencil icon did not visually tell a user that it could be used to create a label and a link.

Consequently, the interface design and interactivity of the tools could be aligned with common software to avoid confusion. For example, students were generally proficient and comfortable with using the chat function in the tasks, which they found to be similar to chatting on Facebook. However, students felt that they should not be restricted to using the chat function to communicate, and preferred to use social networking tools that were more common and authentic, e.g., Facebook, MSN messenger, even mobile phone.

There were some technical issues that surfaced during the trials, such as server failure, long loading times between pages of the tasks, system hanging, and technical bugs in the assessment software, that could have an impact on the delivery of the assessment as well as on students’ motivation and engagement with the activities. While some of these problems could not have been prevented, the lesson learnt is that it is crucial to pre-empt infrastructure problems and conduct comprehensive compatibility testing with the operating platform prior to the implementation of any computer-based assessment.

Challenges

Assessment practices tend to drive learning behaviours and trigger pedagogical responses. For an essentially centralised education system that has achieved strong recognition on the international stage for its students’ academic attainment, a fundamental change in what is being assessed requires substantial efforts. The changes needed are not confined to the professional space but will likely involve social and economic shifts. This represents a key challenge in introducing the assessment of 21st Century skills at the system level.

At the practice level, the tools developed under ATC21S are at best at the nascent stage of an exciting journey. While they point strongly to a reasonable way forward in attempting to automate such assessments, there is still substantial development ahead. For our context, the interpretation of the actions of the students when attempting the tasks will likely need refinement, as such interpretations need to take into account cultural differences amongst others. On top of this, the discourse of the students which is captured could usefully be analysed. This layer of interactions will add richness to the assessment of the students’ 21st Century skills, without which the picture will not be complete. As our understanding of 21st Century skills and their assessment increases, curricula and pedagogical adjustments will need to keep pace to make the collective efforts meaningful and effective.

On the technical level, the key challenge is the ability of the computing system to capture and accurately interpret discourse (including both verbal and non-verbal). Given that natural language processing is still fairly far away from being ideal, and that the ability to interpret human factors such as facial expressions and body language is only at the development stage, a fully automated system for assessing 21st Century skills will not be easy to achieve. A blended approach, at least till the technologies catch up, with teachers being effectively supported by systems that can take up the bulk of the assessment load, could be a reasonable interim practice that helps support ATC21S and similar work.

Conclusion

Most countries, including Singapore, have minimally developed a curricular framework or policy on building students’ 21st century competencies. While there is some extent of agreement on what constitutes 21st century competencies (see for example Voogt and Roblin 2012), there has been less clarity and agreement on how to go about teaching and measuring these competencies. The methods and approaches to doing so cannot be easily extended from current knowledge and the current capabilities of educational psychometrics. They require re-framing as well as new skills and knowledge in the development of tasks and the teaching and measuring competencies. In this sense, the ATC21S international project has made some headway in prototyping what is possible. We are deeply grateful for the honest and constructive feedback that our Singapore teachers and students have given us throughout this project. We take their feedback seriously, as they are the key actors in the teaching and learning of 21st Century skills. We are cognizant of the fact that there is more to be done and look forward to contributing to this important area of work with our partners.