Keywords

1 Introduction

Assessment is an important component in preparing teachers to effectively utilize technology in lesson delivery. The main goal of technology teacher training programs is to equip teachers with the requisite skills needed to utilize technology effectively in their work. Assessment plays a crucial role in assisting to develop these competencies through requiring learners to demonstrate acquisition of specified skills at various stages of the program. This helps to identify learning gaps and the support that students may be needing to master the skills. Traditionally, assessment practices focused on measuring knowledge acquisition, and a letter grade or numerical mark was deemed as an indicator of skill and/or knowledge proficiency or gap (Taras 2005). As such, assessment tasks mainly required learners to regurgitate information provided during teaching (Simonson et al. 2000). The advent of technology in learning introduced a new set of skills that teachers need for effective technology integration. It is crucial to establish how current assessment practices in teacher education are responding to these new requirements.

In Southern Africa, in-service teacher education programs are faced with the challenge of re-training teachers such that they can transform their teaching practices to use technology and equip learners with twenty-first century skills. Lack of preparedness has been reported as one of the leading causes of inadequate adoption of technology by teachers (Hero 2020; Johnson et al. 2016). For in-service teachers, the situation is dire, as for the most part, they are set in their ways and often find it difficult to adopt technology use. In Botswana, there has been little evidence to suggest that technology training for in-service teachers that has been done so far has produced significant results, as most teachers continue to not use technology in their teaching (Kgalemang et al. 2015). Therefore, it is important to study the various aspects of technology teacher training programs to find out how they prepare teachers for technology integration. Such studies may highlight areas where said programs may be lacking and may guide us in developing more effective training programs. In this study, the assessment processes of a technology course in the in-service teacher education program at the University of Botswana were investigated. The overarching research question was how the assessment processes in the course were contributing to the development of technology skills among teachers.

1.1 Context of the Study

The focus of this study was on a course entitled Information and Communication Technology Applications in Schools. The choice of the course for the study was based on its purpose to equip teachers with skills to use technology in teaching–learning processes. The study focused on the way the course assessed students to investigate the role that the assessment was playing in the endeavor to develop these digital skills among the teachers. Traditionally, students taking this course have very low technology competencies, as they received their pre-service training many years ago before technology became part of learning. All the teaching and assessment processes of the course were conducted using digital platforms, mainly Microsoft Teams and Moodle. The course ran for a total of 16 weeks and had previously been delivered using a blended learning approach, but due to the COVID-19 pandemic, the course was offered completely online during this study.

1.2 Significance of the Study 

The results of this study are aimed at improving the assessment processes of technology-training courses to better develop teachers’ skills for effective technology use in lesson delivery. Botswana is venturing into uncharted territories of digital learning, and it is important for teacher training programs to adequately prepare teachers for this new learning landscape.

1.3 Objectives

The objectives of this study were to:

  • establish the students’ technological experiences before they started the course;

  • analyze the assessment activities used in the course;

  • analyze students’ performance in the assessment activities;

  • evaluate the assessment program of the course from the students’ perspective.

2 Theoretical Framework and Literature

2.1 Technological Pedagogical Content Knowledge

The Technological Pedagogical Content Knowledge (TPACK) model by Mishra and Koehler (2006) was adopted in this study to assist in guiding the investigation of the role assessment played in the effort to cultivate appropriate technology skills among teachers. The TPACK Model outlines critical skills that teachers need so that they would be able to use technology effectively. According to this framework, teachers need three sets of knowledge, namely:

Technological Content Knowledge (TCK): This refers to the ability of the teacher to competently use various technology tools in their teaching. The most common resources used for instructional purposes are productivity tools, such as Word processors, spreadsheets, databases, and presentation tools. Online productivity tools have also become an important part of education. According to Johari et al. (2018), developing these skills would help increase teachers’ confidence and comfort with technology. Studies indicate that there is a high correlation between teachers’ technology competencies and their use of technology in delivering their lessons (Anderson and Putman 2020; Dogan et al., 2021). An analysis of literature by Spiteri and Rundgren (2020) to explore factors that influence primary school teachers’ use of technology identified teachers’ skills as one of the critical precursors of technology use. The international disposition of this study suggests that this is a cross-cutting issue that plays a significant role in technology uptake.

Pedagogical Content Knowledge (PCK): PCK entails the knowledge that teachers need to have about their subject matter and how to deliver such content. According to this concept, the different subjects in the curriculum are best delivered using certain specific teaching strategies; as such, teachers need to be skilled in different methodologies and be able to apply them appropriately to achieve their lesson goals. The Rubric for Effective Teacher Technology Use developed by Johnson (2013) recommends that, to better guide the assessment of technology used by teachers, it is important to start with analyzing effective teaching before looking at technology use. This implies that teachers need to have a good pedagogy before they can be good technology users. One constant message that has been preached throughout the years is that pedagogy should drive technology, not the other way around (Attard 2015; Johnson 2013). Technology teacher education programs have a responsibility to ensure that students develop sound teaching practices to help them better integrate technology in their teaching.

Technological Pedagogical Knowledge (TPK): According to this concept, when teachers are knowledgeable and skilled in the various pedagogical approaches that would best deliver their content, they should be able to identify the most appropriate ICT resources that would support such approaches. Over the years, the most common criticism against teachers’ use of technology has been the attempt to impose technology use over traditional forms of teaching which do not support learner engagement and innovativeness (Johnson et al. 2016; Trimble 2003). Promethean (2016) developed an infographic that traces how teaching methodologies have evolved over the years as the technology becomes part of learning. This includes the changes in the dynamics of teacher–student interactions, increased innovation, and increased access to resources, among others. These changes call for innovative and adaptive strategies that leverage the capabilities of available technology resources.

In this study, the TPACK model became a useful guide because, with the requisite skills identified, the framework gave direction on the kind of assessments that should be used in the effort to prepare teachers to use technology. The model was also used as an analytic framework in evaluating students’ performance in the various assessment pieces, thus assisting to identify gaps in the participating teachers’ competencies that needed to be addressed further.

2.2 Assessment in Developing Technology Skills 

Integration of technology in learning requires a synergy of different forms of skills and knowledge as elucidated by the TPACK model, and assessment activities can help identify the areas where teachers may be lacking and direct them to appropriate help (Campbell and Godin 2014; O’Reilly 2016). According to Johari et al. (2018), assessment, if done properly, can assist in developing and honing the needed skills and helps increase teachers’ confidence in using technology.

Performance-based assessment approaches have been identified as one of the best ways to appraise and help improve ICT skills among learners (ITU 2018; Tuparova and Tuparov 2010). This type of assessment requires students to demonstrate their knowledge and skills appropriately through performing a particular activity, either as individuals or in a group. The assessments can take various forms, such as creating a product to use in real life. This kind of activity would require students to outline the procedures they followed in developing the artifact, and educators say it is this journey that is important because it offers a deeper insight into the cognitive and non-cognitive proficiencies of the students (Johnson 2013; Pillai et al. 2020).

Regarding the development of technology skills, performance-based assessment frameworks have been used to measure digital skills, mainly in simulated environments. In these approaches, it is important to first identify the digital competencies to be measured in the assessment (Engelhardt et al. 2021; Tuparova and Tuparov 2010). The assessments are designed to require students to demonstrate a wide range of skills from low level to, most importantly, higher-order ICT skills which require incorporation of other key competencies such as evaluation and critical thinking (Engelhardt et al. 2021). Performance-based assessment designs often adopt a problem-based approach which requires a demonstration of a combination of skills, including technology use, analysis of information, communication, and collaboration (Duch et al. 2001).

In other performance-based frameworks, the assessed ICT skills are separated into operational skills and conceptual skills, whereby learners must be competent in the former before the latter, with the additional benefit of interactivity (Tuparova and Tuparov 2010). This procedure serves a diagnostic purpose because it helps identify areas where the learners need help before moving to the next level and ensures that students master basic skills to ease the acquisition of higher-level ones. Performance-based assessments for measuring technology skills are conducted using ICT resources such as computers and specialized software, and this enables multidimensional testing where various skills can be measured concurrently (Leaser 2020). According to Leaser, this also helps institutions monitor skill development in various areas.

Another important feature of performance-based assessment in technology skills is the provision for self-assessment, which allows learners to reflect on their own capabilities. Studies show a positive relationship between self-assessed technology skills and initiatives to improve on areas of lack, based on feedback (Johari et al. 2018; Sailer et al. 2021). Biggs (2003) claims that the process of learning should equip students with the capability to self-assess their work because, as other authors also indicate, this leads to skilled learners who are able to self-monitor their progress toward achieving their goals (Nicol and McFarlane-Dick 2006; Sadler 1989).

In pursuit of effective ways of assessing learners, performance-based assessment seems to be a better substitute for traditional testing approaches (Quansah 2018; Top Hat 2020; Sumardi 2017). Technology teacher training programs are faced with an enormous task of developing the various competencies that teachers need to efficiently function in digital classrooms. In Botswana, the government is committed to providing teachers with the requisite skills for today’s classrooms, particularly digital skills. Teacher education programs in the country (both in-service and pre-service) are required to provide quality training to help the country realize its aspiration of the Fourth Industrial Revolution (Iloany 2014), and technology training is an important component of these programs. Various studies have investigated these technology-training programs, and one common conclusion has been their ineffectiveness in fully preparing teachers to use technology (Boitshwarelo 2009; Kgalemang et al. 2015). In a study by Major and Tiro (2012), pre-service teachers reported that their training program focused too much attention on theory than on providing them with practical skills. This sentiment has been shared by various other stakeholders in the country, who decry inadequate preparedness of new teachers as they get to the field (Sebobi 2014; Makwinja 2017; Mukhopadhyay et al. 2009).

In reviewing teacher education programs in Botswana, research has mainly focused on the content offered and the teaching methodologies used to prepare teachers (Hunter and Molapo 2014; Iloany 2014; Mangope and Mukhopadhyay 2015; Mannathoko 2013). No study has investigated the way teachers are assessed during their training to understand how it impacts their skills development. Investigating this phenomenon has become even more important in the COVID-19 era in which tremendous changes have been forced upon education. With technology taking center stage in learning, it is imperative to investigate how developing countries like Botswana are adjusting to the prevailing circumstances and are preparing their teachers.

3 Methodology

This research was a case study, and the unit of analysis was the Information and Communication Technology Applications in Schools course at the University of Botswana. The class consisted of 33 students. A mixed-methods approach was used to collect data. Quantitative strategies were used to obtain information on the participants’ background and level of technology skills at the beginning of the course and their evaluation of the assessment processes at the end. Statistical analyses were also performed to examine students’ performance in different assessment pieces to gain an understanding of their skill development during the training. Qualitative approaches, on the other hand, were employed mainly to obtain information on the kind of technology skills that the assessment pieces were designed to evaluate. Open-ended questions were also included in the study surveys to obtain in-depth information about the participants’ needs state and their evaluation of how the course assessments impacted their skills development. Table 4.1 below shows the data collection strategies and the analysis procedures employed in this study.

Table 4.1 Data collection and analysis strategies
Full size table

4 Results

4.1 Students’ Needs Analysis

Table 4.2 shows the age range and gender of the students who participated in the needs analysis exercise.

Table 4.2 Age range and gender of participants
Full size table

Table 4.3 indicates the number of years that the participants had been teaching before they enrolled in the course, juxtaposed with their experience with using computers for teaching. All the participants indicated that they were also holding managerial positions in their schools, such as principals and deputy principals.

Table 4.3 Participants' teaching length and experience
Full size table

The results above show that students enrolled in this course were older people, with all of them over the age of 40. Most of the participants had a teaching experience ranging from 31 to 35 years. 90% of the participants reported that they had no experience in using computers for teaching. The analysis also indicated that, before the course, participants had very low technology competencies, with about 80% indicating that the only things they were able to do with a computer were basic typing and internet surfing. This reported low level of technology skills among these older teachers is a common phenomenon in education, mainly because the technology was neither the part of their teacher training nor the part of learning (Fernández-Cruz and Fernández-Día 2016; Vásqueza and Noriegab 2013).

In terms of access, the study showed that there were no sufficient computers in the schools where the participants worked before taking the course. The data showed that the ratio of the schools’ population to the number of computers available in the schools was very high. The average ratio was 1:32. This implies that there is a need to speed up the pace of provision of computer resources in schools so that the teachers can immediately put into practice what they have learned when they get back to the field. This is crucial because, as studies show, access to resources is one of the major prohibiting factors to technology use in education (Hébert et al. 2021; Johnson et al. 2016).

4.2 Analysis of Assessment Pieces 

Table 4.4 identifies the skills that were evaluated by the various course assessment pieces.

Table 4.4 Assessment pieces and evaluated skills
Full size table

The above table indicates that the three sets of knowledge identified by the TPACK model for successful technology use are addressed in the assessment activities of the course. TCK skills are assessed by requiring students to demonstrate the use of various applications that are used in classroom settings and developing digital artifacts to use as instructional resources. TPK is assessed by requiring students to identify appropriate resources to use in pursuing particular objectives under specific contexts. PCK is assessed through the development of lesson plans and content to demonstrate mastery of the subject matter and identification of the best strategies to deliver specific content. In this study, since the teachers were not able to apply their lesson plans in real-life classrooms, simulated environments, using their classmates as the learners, were used. The peers were also required to give feedback.

4.3 Students’ Performance on Assessment Pieces 

4.3.1 Assignment 1: Development of a Lesson Plan Based on an Instructional Design Model 

Analysis of students’ performance on this assignment revealed that, generally, students performed well in the various components of the lesson plan, with most of the students scoring in the 51–75% and 76–100% quartiles, as shown in Fig. 4.1.

Fig. 4.1
figure 1

Students' performance in assignment 1

Full size image

The area where students seemed to have challenges was the evaluation section; the results show that more than 50% of the students scored in the 0–50% half. Analysis of the students’ work indicated that most students struggled to design innovative evaluation activities that they would use to assess whether the lesson objectives were met. The unsatisfactory performance in designing evaluation activities to assess attainment of lesson objectives illuminated a shortcoming in the teachers’ PCK element as per the TPACK model. Lesson evaluation is an important component of lesson development. Demonstration of mastery of content should be accompanied by robust ways of finding out whether the lesson objectives have been met. Failure to sufficiently address this component means the lesson plan is incomplete (Milkova 2016; Stronge and Xu 2019). Therefore, the study identified this as an area of need in which teachers needed more training.

Analysis of feedback given to students indicated that students were given specific information on what they failed to do in designing appropriate evaluation activities and what they needed to do to achieve this. This is illustrated by the following example:

You need to describe specific activities that you would use to find out whether the lesson objectives have been met; refer back to each objective and identify an activity that can help you find out whether it has been met.

4.3.2 Assessment 2: Part A: Development of Digital Resources

This assignment required students to create a digital resource that they would use as instructional material to teach a particular topic in their subject. The analysis of the students’ performance indicated that most students were able to use applications such as Microsoft Paint and Microsoft Word to develop innovative, appropriate digital instructional resources such as posters and diagrams to be used for their teaching (see Table 4.5).

Table 4.5 Students' performance in the creation of digital resources
Full size table

The analysis showed that students performed very well in terms of demonstrating mastery of applying design principles in developing the digital tools to be used as instructional resources, which improves the effectiveness of the resource (Bandli 2020; Smaldino et al. 2012). The development of skills in producing digital resources is crucial to technology training for teachers. Research indicates that, when teachers are competent in using technology tools, they can identify the best resource to use for their lessons (Dlamini and Mbatha 2018; Drossel and Eickelmann 2017), as TPK also contends. In the African context, it is important that teachers are skilled in developing various digital artifacts so that they can be able to develop material that is contextually relevant to their learners.

Nevertheless, analysis of this assignment also indicated that most students did not do well in terms of identifying the specific objectives they were trying to pursue in their lessons by the material they had developed. The TPACK model says that teachers should have skills in identifying appropriate tools to pursue specific objectives, which means that objectives should be well-defined before the identification and development of resources. In the assignment in this study, students were required to develop the tools first, then identify the objectives that the tools were assisting to achieve. This study shows that this sequencing of skills demonstration is not the most appropriate as objectives must be identified first. Educators advise that is important to first define the goals that are being pursued in education, then identify appropriate technologies and teaching strategies that would facilitate the achievement of such (Kurt 2015; Smaldino et al. 2012).

4.3.3 Part B: Presentations

Here, students were required to do online presentations, demonstrating how they would use the digital materials they had developed in a classroom setting. The presentations were done in groups, and students were also required to assess their peers as they presented. The results indicated that students performed well in the presentations, particularly in demonstrating high levels of confidence, showing that this is a skill they had honed as teachers over a long period (see Table 4.6).

Table 4.6 Students' performance on presentations
Full size table

A shortcoming was noted in the method of delivery where participants demonstrated how they would use the tools in their classrooms. The pedagogical approaches were predominantly teacher-centered, indicating that the teachers needed a lot of training and practice in using learner-centered strategies. According to the TPACK model, for technology to be effective, teachers need to employ appropriate teaching approaches that promote innovativeness and learner engagement to achieve their lesson goals. Rogers (2018) claims that, if a teacher succeeds in delivering their content in an interactive manner using digital resources through their learning management systems, they demonstrate high competence in TCK. Maor (2013) introduces the concept of digital pedagogies, which are described as pedagogical approaches whereby new technologies change the way teaching is done to foster learning. According to Maor, for technology to have a meaningful impact on education, there is a need to shift to this paradigm.

Further analysis of the presentations in this study indicated that students greatly participated in giving feedback to their colleagues on their work and engaged each other. This, according to numerous research, is a vital element of students’ learning because students benefit a lot from peer learning (Duncan 2005; Tulis and Goldstone 2020). Also, involving students in assessment assists them to examine their learning and take ownership thereof (National Research Council 2000).

Another element of note in this assignment was that students did well in designing evaluation activities for their lessons. This component was also in Assignment 1, and students did not do well then. The improved performance in this aspect indicated that students utilized the feedback they were given to better their work. According to Sachdeva (1996), “[e]ffective feedback plays a critical role in helping adult learners achieve their educational goals and reach their maximum potential” (p 1).

Studies based on TPACK indicate that feedback is an important component of skills development in the use of technology. In their 2019 study, Baran et al. found that, in an effort to improve teachers’ skills, there was a positive relationship between increased levels of TPACK and variables such as collaboration, real-world experiences, and feedback. Aktaş and Özmen (2020) conducted a study to investigate the impact of a TPACK development course on pre-service science teachers’ performances. As part of the study, students were required to make a series of presentations based on lesson plans, demonstrating TPACK skills. The results of the study showed that the feedback obtained from prior presentations helped improve performance, including a shift from teacher-centered approaches to student-centered ones. Similar results were obtained by other studies which reported positive gains in TPACK development as students were assessed in various skills and given corrective feedback on their performance (Anderson and Putman 2020; Durdu and Dağ 2017; Tondeur et al. 2019).

4.3.4 Assessment 3: Analysis of the Final Project

This comprehensive assessment required students to demonstrate skills they had acquired throughout the course and apply them appropriately in simulated real-life situations. This mainly involved using different Microsoft Office applications for various purposes in teaching and administration. The analysis indicated that students generally performed well in the various components of the assessment (see Table 4.7).

Table 4.7 Students' performance on the final project
Full size table

The element that seemed to be problematic to the students in this assessment was the development of digital resources. The poor performance in the development of digital resources—particularly concerning the application of design principles in developing the resources—was a surprise, since students had previously done well in this element. This skill was previously assessed in Assignment 2, and students performed well in it. Further analysis of the two assessments revealed that, while in Assignment 2, the assessment question was deliberate in requiring the learners to apply the design principles in developing the tools and specific marks were apportioned to the component, the question in the Final Project did not spell that out, hence they ignored the requirements. This brings to the fore the concept of grading and its effect on students’ learning. Opponents of grading argue that grades are detrimental to students learning because they become an end goal, and students pay more attention to elements that will make them pass without truly internalizing and appreciating what they are doing (McGuire 2015; Miller 2015). This failure of students to apply design principles in this task is a sign that students did not grasp why they needed to do so; they were more interested in the grades. This is concerning, as it may be a sign of what to expect when they get to the field. Unfortunately, this may contribute to the failure to effectively use technology in learning.

It remains a challenge to ensure that students will apply skills learned in the classroom beyond school where no marks are given. Educators and researchers recommend various strategies that could promote students’ understanding and appreciation of learning material for application beyond the classroom. The National Research Council (2000) emphasizes that the skills learned in the classroom must make sense in real-life contexts and teacher trainees must be given opportunities to practice applications of such over an extended period. The TPACK model asserts that, to truly measure the development and attainment of skills, students should be required to address real-life challenges and apply the skills through authentic tasks and document their progress as they show the skills (Mishra and Koehler 2006). In their study on TPACK competencies of Education alumni in their work, Agustini et al. (2019) found that the alumni had higher levels of content and pedagogical knowledge and lower scores in technology application. This gap between technology skills and knowledge gained in learning during teacher training and the application of the skills in real-life classrooms has been extensively noted in the literature (Aktaş and Özmen 2020; Ottenbreit-Leftwich et al. 2010) and needs to be addressed with vigor. Frederick et al. (2006) point out that the classroom is an environment with its own complex mix of factors such as management of learner behavior and other competing responsibilities which may cumulatively tempt a teacher to abandon newly acquired technology skills for the tried and tested traditional teaching. As such, assessment for technology skills also requires training the teachers’ mindset to be able to apply technology in their classrooms, even when the circumstances are not favorable.

4.4 Students’ Evaluation of the Assessments

4.4.1 Perceptions of the assessments

The students reported that the assessment they liked the most was the development of a lesson plan because they found it easier to do. This was corroborated by their good performance on this assignment. On the other hand, they found the development of digital tools to use as instructional resources difficult, because it was their first time doing this. However, regarding usefulness, they considered the Final Project as the most effective, as it allowed them to apply the various skills obtained during the course and helped them to identify their weaknesses, as demonstrated by the following quotes:

…Final Project. It included almost all topics I was taught. The assessment gave me the opportunity to develop and test my ICT Skills I learned in the course.

…Final project. I had to incorporate different concepts which were interesting to do.

According to Aldonio and Andrijani (2008), students’ perceptions of the various elements of their learning are very important, because negative perceptions about certain components lead to failure to perform well in those areas. In the case of assessments, establishing students’ perceptions about the various assessment pieces provided essential feedback in the design of future assessments to capitalize on approaches that resonate well with students of this caliber and generally identify areas that need to be improved.

4.4.2 Technology Use in conducting assessment

Participants indicated that they initially feared the idea of having to use technology to prepare and submit assessments, since their technology skills were low. However, with practice, they ended up being comfortable with this mode and even like it because it helped improve their digital skills further, as shown by the following quote: “Using technology for assignments was excellent, appropriate and quick.” However, some students had misgivings about digital assessments, as they believed that some of their colleagues were using the platforms to cheat: “I feel some colleagues might have taken advantage of virtual testing by copying or not using their own knowledge as some struggled with basic computer skills.”

The students’ feedback indicated that the participants embraced the use of technology in learning and assessments by the end of the course. Consistent with other research, this study showed that older teachers generally fear technology (Raman and Yamat 2014; Winter et al. 2021), but when forced into situations where they have no choice but to use it, they get used to it and find that it is not as difficult as they have imagined. The general attitude toward technology by the end of the course was very positive, which gives hope for extensive use beyond training.

5 Conclusions and Implications

This study showed that assessment plays a critical role in helping develop teachers’ competencies in using technology for teaching. Preparing teachers to use technology should be a comprehensive process that involves the development of an appropriate set of skills needed for effective technology-based delivery. The analysis of students’ performances in the assessment pieces through the TPACK framework assisted in identifying gaps in the development of skills during and at the end of the course, thus assisting to direct appropriate re-mediation and issues to consider for professional development.

The assessments demonstrated that participants in this study, being long-serving teachers, were skilled in traditional aspects of teaching, such as teacher-centered lesson delivery, but needed more help with technology-related skills. This study also showed that certain skills were assessed more than once throughout the course to check how much students had attained them. This process proved to be useful because it revealed the inconsistencies in skill attainment by the participants, particularly when students demonstrated mastery of skill in one element, then later showed a struggle with the same component in a different assignment. This indicated fragility in possession of such skills. Therefore, more effort needs to be put into cementing such competencies among students.

Moreover, the application of acquired skills in the real world remains uncertain. The reported inadequacy of resources in the schools where the teachers work is a cause for concern, because for the teachers to apply said skills more and improve their confidence, they need to practice in their classrooms. Hence, there is an urgent need to provide digital resources in schools. It is recommended that some form of assessment be continued on the teachers beyond the training to check whether they sustain technology use in their work.