Keywords

1 Introduction

Employers are looking for the right workforce for smoother business operations, excellent service, and good workmanship (Lockhart 2013). They usually prefer graduates that are:

  • Comfortable working with others in a team

  • Able to lead and be led

  • Capable of hands-on participation – outside of their typical duties – when necessary

  • Willing to pursue ongoing training and education

  • Sensitive to diversity in the workplace

  • Aware of, and can articulate, personal goals

Hart et al. (1999) describe these skills as integration of both know-how and knowledge-of and extending to personal qualities needed for personal endeavors and new challenges in the workplace. The know-how involves knowledge and development of intellectual skills (Bloom 1956), while the knowledge-of involves the physical movement, coordination, and use of psychomotor skills (Simpson 1972), and personal qualities include the ways in which a person deals with emotions such as change of feelings, values, motivations, and attitudes, among others (Krathwohl et al. 1973). These are clear indicators of the graduates’ expected capabilities upon graduation (IEA 2013). Recent surveys on employers’ need for graduates showed high emphasis on personal qualities compared to technical competence which most graduates lack these abilities (Sternberg 2014; Farkas 2007; Martin et al. 2005). To address this need, universities offering engineering courses are now shifting from the traditional content-based and input-centered method into an outcome-based education (OBE) and output-centered method (Spady and Marshall 1991). In OBE, the focus is on what is most essential for students to know and be able to do upon graduation (Spady 1994). In line with this objective, EAC manual (2012) outlined the OBE process implementation as shown in Fig. 1. As indicated, the OBE process is a top-down approach. PEOs are developed with active participation of stakeholders, advisory committee, and alumni. Attainments of PEOs are best measured 3–5 years after graduation. POs are then developed to achieve the PEOs’ key performance indicator (KPI). PO attainments are evaluated immediately upon graduation to determine whether graduates possessed the required skills to be job ready. To address all POs, LOs are developed in each module based on required skills stated in the POs. Assessments are then developed to measure the LOs of the module by the end of the semester. Mapping of LOs to assessments, POs to LOs, and PEOs to POs is established accordingly. The assessments, LOs, and POs are also mapped to cognitive, psychomotor, and affective (CPA) learning domains. The end product of these assessments is the students’ level of LO and CPA attainments in the module level and the PO and CPA attainments in the program level. Literature showed that most CPA assessments were done in the module level by using surveys (Willey and Gardner 2007; Rovai et al. 2009; Baidowia et al. 2012; Frisby et al. 2014) and peer assessments (Willey and Gardner 2007). Taylor’s University for its part used a quantitative method of assessing the students’ CPA attainments both in the module level and program level using ESAT.

Fig. 1
figure 1

OBE process flow based on EAC manual 2012

Full size image

2 Assessment Methodology

The school of engineering of Taylor’s University offers accredited programs in chemical, electrical and electronic, and mechanical engineering. The school crafted its own program educational objectives (PEOs) and program outcomes (PO) to do OBE assessments and CQI implementation anchored from university core purpose and mission and guided by the EAC 2012 manual (Gamboa and Namasivayam 2014). The university’s OBE implementation model is shown in Fig. 2 (Namasivayam et al. 2013a, b). In the LO loop, the module coordinator prepares the scheme of work (SoW) based on approved PO-LO mapping of the module. The LOs and CPAs are directly mapped to weighted assessments based on SoW. Furthermore, the POs are mapped to LOs, and CPAs are subsequently mapped to both LOs and POs based on CPAs to assessment mapping. This process is carried out in the module level through ESAT (Gamboa and Namasivayam 2012, 2014; Namasivayam et al. 2013a, b). This assessment is performed by the module lecturer at the end of the semester where results are used to evaluate the module’s LO, PO, and CPA attainments based on predefined key performance indicator (KPI) to identify the gains and gaps of CQI implementation and module delivery.

Fig. 2
figure 2

Taylor’s University OBE process flow model

Full size image

A CQI action plan is then prepared to address low LO attainment which subsequently addresses related POs and CPAs to close the loop in the module level. The whole CQI process in the module level is shown in Fig. 3 (Namsivayam et al. 2013a, b).

Fig. 3
figure 3

CQI process flow model

Full size image

In the PO loop, all module ESAT results are stored in the database system to generate the program level PO and CPA attainments. The cohort’s CPA attainments are based on the percent number of students achieving KPI. End-of-semester CPA attainment result can be generated by the program director to evaluate the semester and aggregate performance of students, thus identifying gains and gaps or insufficiency in running the program. CQI plan is then prepared to further improve the program, thus closing the loop on an annual basis and upon graduation of the cohort.

3 Cognitive, Psychomotor, and Affective (CPA) Skills Assessment

The quantitative assessment of CPA attainments begins at the module level using ESAT. Based on approved SoW, PO-LO mapping and the weighted assessment components of the module are loaded into ESAT. The module coordinator then maps the LOs and CPAs to assessment components based on weightage of the assessments. A screenshot is shown in Fig. 4.

Fig. 4
figure 4

LO-assessments and CPA to assessment mapping

Full size image

Assessment components need not be based on 100 as they are automatically normalized to 100. In this process, ESAT automatically provides the mapping of CPAs to LOs as shown in Fig. 5.

Fig. 5
figure 5

PO-LO mapping and CPA-LO mapping

Full size image

Individual student’s marks are then entered into the worksheet for each assessment components, and ESAT automatically calculates the overall marks and the corresponding letter grade for each student. A screenshot is shown in Fig. 6.

Fig. 6
figure 6

PO-LO mapping and CPA-LO mapping

Full size image

Similarly, LO, PO, and CPA attainments are automatically calculated for each student. For example, LO1 is calculated according to Eq. 1:

$$ \mathrm{L}\mathrm{O}1\;\mathrm{Mark}={\displaystyle \sum}\frac{\mathrm{Actual}\;\mathrm{L}\mathrm{O}1\;\mathrm{Mark}}{\mathrm{Acual}\;\mathrm{Maximum}\;\mathrm{Mark}}\times \mathrm{Maximum}\;\mathrm{Normalized}\;\mathrm{Mark}. $$
(1)
$$ \mathrm{C}\mathrm{P}\mathrm{A}\;\mathrm{Mark}={\displaystyle \sum}\mathrm{L}\mathrm{O}\;\mathrm{shared}\;\mathrm{marks}. $$
(2)

A screenshot of the resulting CPA attainments is shown in Fig. 7.

Fig. 7
figure 7

CPA attainment marks for each student

Full size image

In Fig. 7, for each student, a learning domain (C, P, or A) is said to be attained if the student obtains at least 60 % (KPI) of maximum normalized mark. ESAT counts the number of students achieving KPI as indicated in the Figure. Figure 8 shows the bar chart comparing the CPA attainments of previous semester and current semester results.

Fig. 8
figure 8

Comparison of previous and current semester CPA attainment

Full size image

It can be observed in Fig. 8 that current semester’s cognitive and psychomotor skill attainments are higher than that of previous semester attainments. One reason for this is the effectiveness of CQI implementation made in the semester.

In the program level, all module ESAT results are collected and stored in the database system. The cohort’s CPA attainment is calculated based on the percent number of students achieving KPI. Screenshots are shown in Figs. 9 and 10.

Fig. 9
figure 9

Cohort’s CPA attainment main window

Full size image
Fig. 10
figure 10

Cohort’s CPA attainments

Full size image

In Fig. 10, minimum engineering knowledge is said to be delivered by the program if 100 % of the students obtained at least 50 % of all their CPA skills. A second layer is added to serve as target KPI to indicate higher CPA achievements. For each student, CPA attainments are based on the number of modules achieving 80 % KPI. Screenshot of this attainment is shown in Fig. 11.

Fig. 11
figure 11

Individual student’s CPA attainments

Full size image

In Fig. 11, minimum engineering knowledge is considered acquired by the student if at least 80 % of the modules taken achieved 50 in cognitive, psychomotor, and affective learning domains.

3.1 Results and Discussions

The above presentation of CPA attainments used the Electrical and Electronic Engineering program of the university using all modules taken by the students from semester 1 to semester 8 and done by the individual lecturers at the end of the semester using ESAT. In the module level, Fig. 8 shows the comparative CPA attainments between the previous semester and current semester results based on 60 % KPI. Cognitive (C), psychomotor (P), and affective (A) learning skills are considered achieved if the assessment result is not less than KPI. The comparative results will determine the effect of CQI implementation in the current semester. As indicated in Fig. 8, gains were achieved and good practices should be recommended to be maintained. One limitation of this assessment is that attainments of the CPA skills were taken as a whole rather than based on their respective CPA levels. All modules follow similar process and CQI analysis. In the program level, students’ individual attainments for each module were collected and stored in a database which was used to generate the CPA attainments of the cohort as shown in Fig. 10. A minimum KPI of 100 % of students achieving 50 % CPA attainments was set from which the cohort’s CPA should be able to achieve as reflective of acquiring the required minimum engineering knowledge in the program. CQI efforts applied in the program target the CPA attainments to reach at least 80 % of students who achieved 60 % CPA attainments. ESAT was also able to provide the individual student’s CPA attainments based on all modules taken from semester 1 to semester 8 as show in Fig. 11. Again, the result covered only the aggregates of CPA skills rather than the respective cognitive, psychomotor, and affective skills level.

4 Conclusion

This paper presented the quantitative assessment of the students’ CPA attainment result in the school of engineering. Results can be generated for each student and the cohorts at any given assessment period and until upon graduation. Critical evaluation of these results will reveal a lot of information on the strengths and weaknesses of the quality of teaching and student learning experience through the years. Gaps and insufficiencies such as breadth and depth of curriculum, concerns on module delivery and assessments, poor skill achievement of students, and staff capabilities among others are major contributory factors of students’ inability to attain the target KPI for cognitive, psychomotor, and affective learning skills. With this in place, CQI action plan and its implementation can be done immediately to enhance the quality teaching and the student learning experiences in the program. The results however covered only the aggregates of the CPA skills rather than their respective skill level. More meaningful results could be derived if assessments can be broken down into the respective skill level, say, how many students achieve level 1 to level 6 of cognitive, psychomotor, and affective skills.