1 Introduction

This chapter describes a new approach to competency assessments within work integrated learning (WIL) to readily measure and depict changes in competence over time. The recognition of competency assessment, in place of aptitude or intellectual ability as a predictor of performance, originated in the 1970s (Mirabile, 1997). Within accredited Australian organisational psychology postgraduate programs, students attain competency across a number of consecutive WIL placements, in combination with coursework and a research project. The areas of competency to be attained during the WIL component of these programs include seven core competencies specified by the Australian Psychology Accreditation Council (APAC, 2010). These competencies were designed to ensure students attain the minimum level of competence necessary for full registration as a psychologist.

Given the breadth of areas of practice within the field of organisational psychology (e.g., recruitment, change management, coaching, training/facilitation, employee wellbeing), students undertake placements within a wide range of organisations, within an equally wide range of contexts (public and private sectors, large and boutique consultancies, etc.). Student performance across the core competencies is assessed by their assigned supervisor, who may be either internal or external to the placement organisation. A student will have a number of supervisors across their WIL placements. For this reason, it is important that the way students are assessed is robust, accurate, consistent and representative, thereby minimising the subjective bias of individual supervisors as much as possible. The application of competency-based assessment models to environments of successive yet independent learning, such as WIL within organisational psychology, has not been undertaken.

To capture performance across successive placements, a trajectory tool was developed depicting areas where students hold key strengths (i.e., higher competence scores across all placements), areas where they have acquired strengths (i.e., progressed from low to high competence across placements), areas where they have wavering strengths (i.e., a mix of high and low competence across placements), and areas where they need to continue to grow (i.e., lower competence across all placements). The tool enables students’ competency development to be tracked across the breadth of their WIL placements, providing students and supervisors/managers with evidence of changes in competency over time, which can inform the assessments of a student ’s suitability for the transition to employment, while also identifying areas for professional development post-graduation. This approach to placement assessment provides the students with feedback on their practicum assessment, facilitating the enhancement and improvement of their professional learning strategy.

2 Context – Organisational Psychology in Australia

Organisational psychology is a specialist field that examines the psychology of work. The field incorporates a number of disciplines including coaching, consumer psychology, ergonomics, human factors, human resource management, industrial psychology, managerial psychology, occupational psychology, personnel psychology, vocational psychology, and work or business psychology (Australian Psychological Society, 2019). Consequently, the types of roles that Organisational Psychologists may specialise in are equally as varied, including recruitment and selection, learning and development, leadership and talent management, coaching, mentoring and career development, change management, evaluation and workplace research, occupational health and safety, performance management, wellbeing, stress and work-life balance (Australian Psychological Society, 2019). Recent examples of the types of activities undertaken for organisational psychology WIL at Griffith University are representative of the varied nature of the field, including designing recruitment and selection protocols, interpreting and feeding back job applicant psychometric assessments, undertaking usability assessments, delivering team building interventions, commissioning a training needs analysis, delivering learning and development sessions or facilitating training, delivering corporate coaching, implementing program change and evaluation, and undertaking culture interventions such as change management initiatives, safety culture or equity and diversity initiative enhancements. Given this diversity in tasks, assessing competence is complex, especially when different competencies are utilised for the various types of placements, and students are not expected to undertake WIL in every facet of organisational psychology practice. To this end, performance on WIL activities is typically assessed against a generic set of competencies for psychologists. The organisational psychology program at Griffith University assesses students on the seven APAC (2010)Footnote 1 core competencies:

  • Knowledge of the discipline

  • Psychological assessment

  • Intervention strategies

  • Research and evaluation

  • Oral communication skills

  • Written communication skills

  • Ethical, legal, and professional matters

Graduates from these accredited programs are eligible for full registration with the Australian Health Practitioners Regulation Agency (AHPRA), entitling them to call themselves a Psychologist. As Psychologists, these graduates have the safety of the general public as their primary mandate (AHPRA, 2019). For this reason, it is imperative that graduates are accurately assessed as meeting the prescribed competencies to practice as a Psychologist.

Within Australia, postgraduate students undertaking their 5th and 6th years of study in an accredited psychology program (e.g., Master of Organisational Psychology, Master of Clinical Psychology, Master of Forensic Psychology) are required to undertake at least 1000 h of WIL and demonstrate competency across the seven core competency areas specified in the APAC standards. The types of WIL tasks, logs of WIL activity, type and frequency of WIL supervision are prescribed and routinely audited by APAC. For example, WIL students are required to undertake their placements in a minimum of three locations, undertaking 1 h supervision for every 7.5 h of activity, with more than 50% of their supervision needing to be individual and 70% of their supervision being provided by a supervisor who is internal within the placement organisation.

Given the integrated nature of WIL within these accredited postgraduate psychology programs, it is no surprise that WIL typically constitutes 40% or more of the postgraduate program. Consistent with the recruitment literature, WIL acts as a realistic job preview, which we know to be the best predictor of on-the-job performance and therefore graduate employability (Chehade & Hajjar, 2016). For this reason, it is important that performance is accurately assessed within each WIL experience, as well as across successive WIL experiences.

3 Assessing WIL Competence

The significant impact that psychologists can have on individuals and organisations clearly articulates the importance of accurately assessing the competence of students prior to graduation and subsequent registration as a psychologist. At Griffith University, the WIL experience is undertaken through a series of placements within a minimum of three different organisations. Students enrol in practicum courses, with the associated workload for these courses estimated to be in the vicinity of 1100 h – attending practicum classes, undertaking 1000 h of placement activity in addition to supervision, as well as documenting and reflecting on the placement journey through completion of the necessary case-notes and paperwork. Given the large proportion of the program that is dependent upon successive WIL performance, it is imperative that the mechanisms used to assess competence are reliable and valid, while also being suitably robust to adapt to the varying nature of organisational psychology placements, given the breadth of this field of practice.

Assessment provides evidence of a student’s capability or outcomes from WIL , and should also help students understand their own learning. However, Yorke and Vidovich (2014) argue that assessment practices in WIL have lagged behind developments in the provision of WIL . The challenge for assessments of WIL is that the full range of specific observable work actions or behaviours cannot be assessed. Yorke (2006) argue that we need to move from standardised grading systems, to an intrinsically personalised component that assesses how the individual uses the knowledge, skills and judgement associated with the profession to perform effectively in the domain of possible encounters defining the scope of professional practice. Furthermore, Hodges, Eames and Coll (2014) argue that the assessment of student work placements has been particularly problematic because the work practices in these settings are inherently interactive, collegial and interdependent, and involve hard and soft skills and a range of assessors.

The successive nature of placements in postgraduate psychology programs is often at odds with the typical assessment mechanisms embedded within many higher education institutions for these professional education programs. The enrolment in courses or units of study that are individually passed hides the dependent nature of these WIL activities and the inherent building of competence that is integral to those programs with large WIL components. Thus, we need an individualised/personalised approach to assessment that moves beyond disconnected assessment, to a process where feedback not only assesses current performance in a placement (summative assessment) but informs future development in subsequent placements (formative assessment). This developmental focus should extend beyond formal learning, with Boud and Falchikov (2006) arguing that assessment activities should not only address the immediate needs of certification or feedback to students on their current learning, but also prepare them for lifelong learning.

We identified that improvements were needed in the way in which student competence was being assessed in organisational psychology WIL placements, with the current method of assessment failing to provide students with the opportunity to track their competency development over their consecutive placements. This meant that each placement was assessed and regarded independently, rather than as a successive trajectory of competence development that results in greater insight into student strengths. The purpose of engaging in this project was to identify a methodology for evaluating student experiences in placements that could be transferred to their successive placement experiences, and eventually transferred to their future workplace post-practicum, thereby enhancing graduate employability and informing ongoing professional development.

4 Our Approach to Developing a Competency Assessment Framework

We wanted the evaluation tool to be able to be used to gauge student capability at the start of a placement, and then re-assess student capability upon completion of each placement . The ability to reliably and validly assess transitions in student performance as an individual placement progresses enables students to better identify their areas of strength and development, maximising the learning opportunities for students during each individual placement experience. A better understanding of the student ’s skills and expertise during placements will also enable the student to identify what type of work is best suited to their competency strengths, potentially enhancing future job satisfaction and performance.

We also wanted to identify a competency assessment framework that could be used by consecutive supervisors to assess student development over sequential placements. The resulting competency assessment tool should facilitate the supervisory relationship and enhance each placement experience, by linking performance on successive placements and thereby building a trajectory of competency development. Students can be guided toward choices that either extend strengths or areas where they need to grow, rather than repeating WIL in areas that are already well developed. Such a tool would also identify areas for continued professional development and life-long learning, in line with the mandated requirement of maintaining psychologist registration after graduation (Psychology Board of Australia (PBA), 2015).

It is well documented that any tool assessing competency should meet the criteria of validity, reliability, feasibility and acceptability to all stakeholders. According to Masters and McCurry (1990) within the field of qualitative assessment: validity is often used interchangeably with accuracy, and reflects whether the assessment is measuring what it claims to be, so as to achieve its intended outcome; reliability is often used interchangeably with consistency and measures the extent to each assessor uses the same performance cues when making their ratings, or making assessments of the same competence across difference assessment methods; feasibility refers to the assessment being realistic and practical to implement while imposing manageable demands; and acceptability refers in part to perceived fairness, in addition to the tool being perceived as providing value to the individual being assessed, including feedback on their level of competence, and actions to be undertaken in areas where competence has not yet been attained. After reviewing the research on recommended competency models, three models for assessing competency were evaluated by our participants: the current Likert rating scale, a Pass-fail rating scale and Miller’s pyramid model. Briefly, these models involve:

  1. 1.

    Likert rating scale: Likert scales help to establish the importance of a particular competency, the proficiency level for each competency, and the level of competence demonstrated by an individual. However, they tend to produce ratings that cluster around the middle or above the middle of the scale range, a central tendency bias (Albaum, 1997). Likert rating scales are limited in how they can differentiate performance levels. The current placement competency evaluation tool utilised a Likert rating scale.

  2. 2.

    Pass-fail rating scale: Despite criticism that a pass-fail grading model results in students reducing their effort to the minimum level required, numerous papers mitigate these concerns by presenting evidence that after implementing a pass-fail grading model, students consistently did not decrease their effort or motivation (Friemuth, 1970). Three rating options are available – pass, fail, and not assessed. The Pass-fail rating scale integrates the assessment of competencies with the current Griffith University organisational psychology postgraduate placement course grades, which also adopt a pass-fail approach, awarding either a ‘non-graded fail’ or a ‘non-graded pass’. This model held merit as it would streamline course gradings.

  3. 3.

    Miller’s pyramid model: The framework for assessing competence proposed by Miller (1990) involves a scale of competence ranging from different levels of ability (knows; knows how; shows how; does). The model embraces transfer learning theory, while emphasising the role of reflection in allowing students to put theory into practice and transition their skills successfully across different contexts (Yashin-Shaw, Buchridge, Buckridge, & Ferres, 2004) Reflection is recognised as a critical component of the psychology profession with mandated peer consultation a continuing professional development requirement (PBA, 2015).

A total of 59 key stakeholders were contacted and given the opportunity to participate in the interviews, of which 33 participated. Of the 33 participants, 17 were students who had been actively enrolled in at least one practicum course in the period of 2016 to 2017 and 16 were supervisors who had actively supervised at least one Griffith University student in the period of 2016–2017. The research team received approval from Griffith University’s Ethics Committee (2017/522) to conduct the evaluation. Interviews were conducted from September to October 2017, with both students and supervisors. Students who were provided the opportunity to participate were at different stages in the postgraduate program, ranging from their first to last placement . Additionally, students could be enrolled in either a Master of Organisational Psychology and PhD (Organisational Psychology) program. The participant pool was not inclusive of postgraduate students prior to 2016 as a review undertaken in 2015 resulted in a new approach to placements being implemented in 2016, so the applicability of their comments would be limited. Supervisors were from a range of different organisations offering a variety of different placement opportunities. Supervisors interviewed were approved by PBA to provide placement supervision, with 50% of interviewed supervisors also holding endorsement in organisational psychology.

Semi-structured interview questions explored participants’ perceptions and experiences of the end-of-placement evaluation rating scale currently implemented in our organisational psychology placement courses. Participants were also presented with the two alternate models, described above, and asked to identify the strengths and weaknesses of each. Sample interview questions included:

  • How should competencies be addressed?

  • What would be strengths of each measure presented?

  • What would be limitations of each measure presented?

  • Is there room for improvement of the presented measures?

Interview notes were analysed using thematic analysis (Braun & Clarke, 2006). The data was examined separately for each of the three models, and then the comments for students and supervisors were compared for each model. These findings will now be presented.

5 Findings

In general, the student feedback centred on needing a competency measure that would help them to learn and enhance their skills as an Organisational Psychologist. As was anticipated, students sought a measure that would provide them with a trajectory for progression and continual development. Three key perceived value additions of an enhanced WIL competency assessment tool emerged, namely:

  1. 1.

    To secure feedback on a student’s workplace experience;

  2. 2.

    To inform choices about career, work options or specialisations; and

  3. 3.

    Identification of how these experiences can increase a student’s employability.

Aligned with these findings, students also indicated a strong preference for the timing of any post-placement intervention to continue to be after each placement experience, with the potential to provide an indication of competency development over time (across their three or more WIL placements). In contrast, supervisor feedback centred on a measure that would be interpreted equitably by all supervisors and would not impose a significant workload.

5.1 Seven Key Themes

Examining the data across participants, thematic analysis (Braun & Clarke, 2006) revealed seven key themes (see Table 1): validity, inter-rater reliability and subjectivity, social desirability, sufficient detail and range, usability, not assessed component, and constructive feedback. While themes were discussed by both students and supervisors, as detailed below (see also Table 2), there were both similarities and differences in their views. During the interviews, both students and supervisors were asked to compare and contrast the three proposed assessment models (Likert ratings scale, Pass-fail rating scale, Miller’s pyramid model). These comparisons will now be presented for each of the seven themes.

Table 1 Definitions of thematic analysis key themes
Full size table
Table 2 Summary of feedback from students and supervisors on each assessment tool
Full size table

Validity

Students and supervisors were vocal regarding the need for high validity of the proposed model, with the model needing to demonstrate the incorporation of a research or evidence based approach to practice (the scientist-practitioner approach), measuring more than just performance. This may include examining the overall suitability of the student for that area of the profession. In general, both students and supervisors strongly believed that Miller’s pyramid model demonstrated a scientist-practitioner approach.

With regard to the Likert rating scale, student opinions were divided with 50% of students stating that the Likert rating scale measured what it intended to measure, namely organisational psychology postgraduate practicum competence, and the other 50% commenting that the Likert rating scale was very limited in that it did not provide a rigorous scientist-practitioner approach. Students argued that the use of the words ‘satisfactory’ and ‘competent’ as separate levels is not accurate stating, “there is confusion with rating 2-satisfactory, the wording is inappropriate as this is technically saying the same things as competent.” Supervisors expressed concern that the Likert rating scale merely measured students’ performance instead of competency. Supervisors also stated that although the Likert rating scale does acknowledge the student ’s stage in the course, there are still grey areas making the scale ambiguous and less than ideal.

Neither students nor supervisors commented on the validity of the Pass-fail rating scale. Given the nature of the comments from students and supervisors regarding the alternative models, this lack of comment may be interpreted as there not being any perceived issues with the validity of this scale.

Students identified Miller’s pyramid model as being capable of distinguishing the difference between students’ knowledge and behaviour, as it measures a student ’s ability to take theory and apply it to practice. Supervisors felt that Miller’s pyramid model explicitly focused on evaluating a student ’s behaviour. Of the supervisors who commented, 75% commented positively about Miller’s pyramid model as it aligned with the values of the organisational psychology profession and the scientist-practitioner model. However, one supervisor argued that the model was confusing as ‘knows’ doesn’t necessarily precede ‘shows how.’ However, the WIL focus on reflection and developing student ’s self-awareness is beneficial in addressing this limitation.

Inter-Rater Reliability and Subjectivity

Overall students and supervisors both favoured Miller’s pyramid model for assessing inter-rater reliability and assessor subjectivity. Stakeholders believed that Miller’s pyramid model would facilitate conversation, as it defines elements of practice. It was believed that Miller’s pyramid model would demonstrate further objectivity with the inclusion of more detail and clarity for each anchor.

Of the 17 students who participated, 70% believed that the Likert rating scale was too subjective. Students believed the descriptor of ‘given the student ’s stage in the course’ was extremely subjective as it “asks supervisors to make judgement calls based on (the supervisor’s) experience”. Of the 16 supervisors, 75% commented on issues relating to subjectivity and inter-rater reliability. The majority of supervisors agreed that the descriptor, ‘given the students stage in the course’, was very subjective and relied on a supervisor’s definition of what the student should be achieving on each placement . Supervisors highlighted that each student is different and encounters different placement experiences. A student without industry experience can be expected to have very different abilities and skills compared to a student who has returned to university after 20 years of industry experience. Hence, these students would perform at different levels, even if they were at the same stage in the course.

Students were not as concerned about issues with subjectivity and biased ratings in the Pass-fail rating scale. Similarly, supervisors were not as concerned about issues with subjectivity and biased ratings in using the Pass-fail rating scale, however, supervisors did question where the normative point was.

Students reported that Miller’s pyramid model removed some subjectivity as it would facilitate a collaborative conversation. This conversation would facilitate the supervisor and student reaching agreement on the student ’s rating, thereby assisting student acceptance of the assigned ratings. Supervisors reported Miller’s pyramid model as being more objective than the other models, as it defines elements of practice. There were, however, concerns around poor inter-rater reliability due to the lack of clarity and definitions of what constitutes each of the knows, knows how, shows how levels for each of the psychology competencies.

Social Desirability

Supervisors commented on the social pressure they feel to score a student highly stating, “[they] don’t want to give a two or below due to social desirability, as it flags a negative connotation.” This finding is not surprising as postgraduate students are high achievers (having attained high grades in their undergraduate studies to progress to Honours, and then high grades in Honours to progress to post-graduate studies). Moreover, there is a general tendency for work placement supervisors to give inflated marks across capabilities (Jackson, 2018). Students and supervisors agreed that the use of a numerical rating offered no value.

Students commented on the negative focus of anchors in the Likert rating scale, highlighting the negative impact of social desirability on a student’s professional development and how this is not constructive for the student ’s development (e.g., receiving a rating of a ‘2’ on an advanced placement ). However, one student felt as though there were no negative connotations associated with receiving a lower rating. Supervisors stated that they feel as if they cannot rate a student as ‘2’ due to social desirability, as it attaches a negative connotation. Adding to this, supervisors consistently mentioned the irrelevance of numerical annotation stating that the numbers are unrealistic and merely buy in to social desirability issues. Further resulting from issues of social desirability, supervisors reported a perception that students get upset if they are not assigned a majority of ‘5 s’, however, supervisors commented that the ‘performance equal to or above that of a fully competent professional’ rating is too extreme, and they would rarely give students a rating of ‘5’. Further, some supervisors did not feel that the ‘5’ rating was achievable for any student , although it should be noted that some students are already fully registered psychologists, so attaining this rating is, in fact, feasible. Supervisors also felt that it is hard to identify and define a ‘fully competent professional’. Supervisors highlighted issues with all of the anchors being positively positioned, such that this can be undermining for students.

Students and supervisors favoured the Pass-fail rating scale and Miller’s pyramid model, as these models implemented positive and growth focused wording, mitigating issues of social desirability. Students liked that the Pass-fail rating scale did not incorporate a numerical rating system and felt that this helped to mitigate issues of social desirability. Supervisors also made positive comments for both these models due to the absence of numerical ratings, reflecting that “not being numbered forces people to think hard about what they are rating.” For the Pass-fail rating scale, supervisors highlighted the great use of the terminology ‘yet’ in ‘not yet competent’ and ‘strength’ as opposed to ‘expert’ in mitigating social desirability issues. However, a minority of supervisors felt that it would go the other way and would be harder to rate a student as ‘not yet competent’ rather than the currently used anchors of ‘performance below expectations’ and ‘satisfactory, but would benefit from more experience’.

Within Miller’s pyramid model, students liked the use of the descriptor ‘developing proficiency’ and believed that this softer language helped to mitigate social desirability issues. Students made comment on the absence of the numerical ratings, stating that this was a strength of the scale. Supervisors were not very vocal regarding Miller’s pyramid in relation to social desirability. However, of the supervisors who did comment, there was agreement that the use of softer language and the positive positioning of the descriptors was a strength of Miller’s pyramid.

Sufficient Detail and Range

Overall, students and supervisors agreed that there was no benefit to the breakdown of levels indicating ‘not competent’ within the Likert rating scale. The Pass-fail rating scale addressed these concerns and included an appropriate three level breakdown of competency, however supervisors held some concerns regarding the definition of competency. Miller’s pyramid model was believed to lack clarity and detail in each of its descriptors.

Overall, 70% of the student comments in relation to the Likert rating scale were positioned negatively, indicating that students believed the range was not optimal and the scale descriptors contained insufficient detail. They critiqued the use of two not yet competent anchors, in addition to a lack of specific detail in the descriptions of each anchor. Students did not believe the descriptors were able to effectively capture their individual differences over and above their course stage, yet this may be relevant when making competency assessments. Supervisors echoed the redundancy of the two levels of not competent asking, “what is the point of having a 1–5 range when APAC only require students to meet competency?”. Supervisors also stated that the overall scale range was not used appropriately by supervisors due to social desirability issues, as previously discussed. In contrast to the students, however, supervisors believed that the level of description of the anchors was adequate and that the use of the wording, ‘given the student’s stage in the course’ provides a normative point for supervisors. Interestingly supervisors did comment on the presence of a competency range, when accreditation requirements only necessitate a basic level of competence being met.

Students felt that the Pass-fail rating scale labels were less daunting, and the scale followed a clear, step-wise approach. However, although the three-level breakdown of competency was well received by students, there was still concern around the context of competency. Students stated, “[It’s] not clear what you are competent against, there is no clarity around what that means and your stage in the course.” Multiple students commented that the Pass-fail rating scale did not provide a clear definition of competency. Supervisors responded positively towards one level of ‘not competent’. However, there was again contention among supervisors as to the divide of competency into three distinct levels. Of the supervisors who commented, 50% agreed that dividing competency into distinct levels was a strength of the Pass-fail rating scale, and 50% felt that it was redundant. Supervisors agreed that the scale needed to define what competency is. A handful of supervisors commented on the redundancy of the descriptor, ‘needs future development’, as professionals in our discipline always require professional development as mandated by the PBA (2015).

Students reported that Miller’s pyramid model lacked significant detail and explanation to be utilised effectively by supervisors, with supervisors affirming, “[they] would need training because there isn’t enough detail”. There was concern with ‘knows how’ being the equivalent of competent if a student only knows how to complete a task but doesn’t actually show how they engage in the respective behaviours. However, other students felt that Miller’s pyramid model clearly breaks down student performance. There was some confusion around the terminology used, with specific mention of ‘global development’. Students were unclear on the definition of global development and how to identify a student who was demonstrating this level of competence. Supervisors also described Miller’s pyramid model as vague and ambiguous, specifically with regard to ‘global development’. Miller’s pyramid model was described as lacking detail and clarity, however positively, it did maintain a practical focus.

Usability

Students and supervisors agreed that the recommended model should follow a simple, practical and user-friendly format. Students stated that the Pass-fail rating scale and Miller’s pyramid model conformed to these guidelines the best, in contrast the supervisors believed that the Likert rating scale and Pass-fail rating scale worked best.

Students highlighted the complexity of the scale descriptors used within the Likert rating scale assessment tool, with more than 50% of student comments alluding to the onerous and lengthy scale descriptors. Less than 50% of students believed the current scale was simple in layout, visually appealing, and easy to use. Supervisors did not express concern relating to the usability of the Likert rating scale. All of the supervisor’s comments were positive in nature indicating that the Likert rating scale is useable, for example, “the scale works, it’s comfortable and usable.”

Students agreed that the Pass-fail rating scale offered a user-friendly approach, with 70% of students responding positively to the usability of the scale. Students highlighted the real-world and practical approach that the Pass-fail rating scale offered, adding that it is readily useable, as it incorporated fewer words and simple language. Supervisors agreed that the Pass-fail rating scale was user friendly, with 56% of supervisors reporting a high degree of usability. Supervisor’s comments indicated that the Pass-fail rating scale was intuitive and easy to use, as it made great use of a simple layout and simple language.

Students highlighted the usability of Miller’s pyramid, with the concise descriptors, softer language, and a practical approach, reducing the demands on the supervisor. There was some minor concern from students that the model was too informal and mimicked that of a school grading system. Contrarily, students suggested improving engagement with the form by using a visual aid, such as the pyramid from the model, in place of a typical table. Supervisors mirrored student comments by also highlighting the use of softer language and the practical application of the scale. Additionally, supervisors commented on the potentially increased time burden on supervisors when using this scale.

Not Assessed Component

There was strong consensus between students and supervisors that the proposed model needed to incorporate a ‘not assessed’ component. This could be a comments box, similar to that currently used, however, the process for scoring ‘not assessed’ competencies must be standardised across supervisors.

Students felt that supervisors were forced to give a rating for every competency on the Likert rating scale, yet not all WIL placements provide students with exposure to the entire range of competencies. Students address competencies collectively across their successive placements, as opposed to on each individual placement . It is therefore of no surprise that 59% of students believed that the scale needs to hold provision for a ‘not assessed’ component. Supervisors as a majority did not comment on a ‘not assessed’ component for the Likert rating scale. In discussing this issue with the Placement Coordinator, it became apparent that supervisors vary in their use of the ‘not assessed’ option within this model, with some supervisors using the comments section and not providing a rating, and other supervisors addressing the competency throughout supervision even if it was not related to the actual WIL activities, thereby enabling them to make an informed rating of the student ’s competence.

Students indicated that the inclusion of a ‘not assessed’ component was a strength of the Pass-fail rating scale. However, students did not show awareness that the inclusion of a ‘not assessed’ component will result in having to ensure there is evidence across placements collectively to suggest all competencies have been assessed. Supervisors reported addressing a competency, which a student may not have had the opportunity to demonstrate given the particular project or placement constraints, in a range of different ways. Supervisors were in favour of a standardised approach to addressing competencies which are not assessed on given placements or projects. A minority of supervisors believed that there was no value to a ‘not assessed’ component, with one supervisor commenting that a “not assessed option wouldn’t be used that often as although competencies aren’t performed on every placement , they are still all usually discussed during supervision sessions.”

Students agreed that the Miller’s pyramid model needs to incorporate provisions for competencies that are not assessed during particular placements. This could be ensuring there is a comments box for supervisors to provide justification, as outlined in the above discussion with the Placement Coordinator.

Constructive Feedback

There was a disconnect between students and supervisors regarding the value of constructive feedback. Students strongly believed that there was an immediate need for the proposed model to provide constructive feedback to students to inform their placement trajectory and student employability. In contrast, supervisors believed a traditional pass-fail approach was sufficient, as required by APAC and the University. Students identified the benefits of a scale that can track their development and provide future direction for employability. Although onerous for supervisors, students made comment on how helpful the rating is when also provided in conjunction with comments to support the rating. These findings are consistent with research on how university students’ and staff perceptions of feedback differ (Orsmond & Merry, 2010; Savin-Baden, 2010). For example, students believe they are receiving less detailed feedback than staff do and value comments, while staff believe students are more focused on marks or grades than how to improve their learning (Carless, 2006).

The Likert rating scale assessment of ‘3, competent (given the student’s stage in the course)’ may represent development, but also may not. The scale introduces a large amount of ambiguity when it is reviewed and interpreted in isolation. Adding to the ambiguity, many supervisors do not provide comments to justify their ratings on each competency. Students also identified a rating of ‘5, performance equal to or above that of a fully competent professional’ as concerning as it can be polarising, and students shouldn’t be expected to be at this level. Supervisors did not express a need for the model to inform a student ’s future direction or strength. Supervisors discussed the strengths and barriers to a student ’s motivation of an extreme score, either high or low. A high rating could inhibit a student ’s motivation for growth and development and low ratings (e.g., 1 or 2) can potentially demotivate students.

Student feedback regarding the use of the Pass-fail rating scale was positive, with 65% of students commenting on its effectiveness and its ability to track growth and development, “[it] provides a coaching tool for supervisors to provide feedback.” Supervisors agreed with the student ’s comments; however, a minority of supervisors were concerned that the Pass-fail rating scale lost an element of helpful feedback as the anchors were not clearly operationalised in an organisational psychology context.

Students commented on the ability of Miller’s pyramid model to facilitate deeper reflection. They believed that this model will help to increase a student ’s self-awareness through facilitating collaboration, communication, and, as a result, deeper reflection . Supervisors added to this, concluding that this model creates a coaching space to help students have clarity and confidence in their skills and abilities. Supervisors responded extremely positively to Miller’s pyramid model, including that the model, “allows facilitation between students and supervisors and encourages a coaching like space,” as well as, “[it] provides students with future direction.”

Summary

Overall, the stakeholder feedback across the seven themes identified that Miller’s pyramid model was preferred, followed by the Pass-fail rating scale. Miller’s pyramid model received the most positive feedback in relation to validity, inter-rater reliability and subjectivity, as well as constructive feedback, while also being preferred together with the Pass-fail rating scale for social desirability. No preferred assessment tool was identified for usability, while the Pass-fail rating scale was preferred for both sufficient detail and range, as well as including a not assessed component. In light of these findings, the decision was made to proceed with Miller’s pyramid model as the basis of the new competency assessment framework, with adaptations to address the concerns raised by participants.

6 New Competency Assessment Framework – Evaluation Tool

On the basis of the analysis of the interview data, an adapted version of Miller’s pyramid model was developed. The new competency evaluation tool (see Table 3) was trialled through a ghosting phase, with supervisors completing the existing competency assessment tool concurrently with the new evaluation tool. In addition, for students commencing placements, supervisors were asked to complete the tool at the start of the placement (approximately 1 month after commencement), as well as at the end of the placement . This provided an indication of the likely change in competency development within an individual placement .

Table 3 Assessment rating criteria for each postgraduate organisational psychology areas of competence
Full size table

7 Validation

In total 6 supervisors, supervising a total of 15 WIL placement students, participated in a trial of the new competency evaluation tool. Supervisors were asked to rate the student ’s competence early in the placement and again at the end of the placement . For some students, due to the fact that the student had already commenced their placement when the trial began, supervisors made these ratings at the same time, thus rating where they believed the student was at the start of their placement . Analysis of the data demonstrated variation in competency assessments over time within WIL placement , regardless of which method of assessment (sequential or concurrent) was undertaken.

Student ratings changed over the course of their placements. These differences over time demonstrated that assessments in competence varied from the start to the end of a placement . It should be noted that no student demonstrated an improvement on every competency during a placement . This finding is important as it demonstrates that the supervisors who participated were not susceptible to a bias in their assessments, such as a halo bias, where all competencies would receive a similar improvement from the beginning to the end of a placement . Interestingly these within placement changes were both positive and negative, namely, both increased competence and reduced competence. These negative changes may indicate that initial assessments of students were not accurate, or that the student demonstrated less competence as their placement progressed. Anecdotal discussions with students suggest that their awareness of these changes over time was not always apparent. As such, the new competency evaluation tool provided additional information to students in relation to their performance. To this end, the longitudinal use of the tool was identified as being highly beneficial.

Anecdotal feedback suggested that the new competency evaluation tool provided students with new and unique information they had not previously gleaned through the supervision process. For some, this was information in relation to which areas of practice they would like to pursue for their career. We suggest that providing students with better guided career options facilitates employability post-graduation, which is a key performance indicator for most academic institutions (e.g., Griffith University, 2017).

Both supervisor and student feedback on the assessment ratings and descriptors within the new competency assessment tool were also positive. This indicated that the new tool was prima facie meeting the seven themes for assessment identified during the initial phase of the project.

8 Competency Trajectory Tool

To facilitate student competency development as a trajectory across their placements, a new visual depiction of competency acquisition will also be implemented. Utilising a radar chart design (see Fig. 1) the competency trajectory tool will be used by students successively over each of their placements to demonstrate their competency development throughout their program of study.

Fig. 1
figure 1

Example display from the competency trajectory tool

Full size image

The competency trajectory tool demonstrates to students the areas where they possess key strengths (i.e., higher competence across all placements; e.g., Knowledge of the Discipline in Fig. 1), the areas where they have acquired strengths (i.e. progressed from low to high competence across successive placements; e.g., Written Communication Skills in Fig. 1), the areas where their performance is inconsistent (i.e., a mix of low and high competence; e.g., Psychological Assessment), and the areas they need to continue to grow or focus their professional development (i.e., lower competence consistently across all placements; e.g., Ethical, Legal and Professional Matters in Fig. 1). This methodology enables students to develop a guided learning strategy for implementation post-practicum.

The competency trajectory tool also enables students to not only reframe their approach to placements as a series of enhanced skill acquisition opportunities, but to also see the areas of psychological practice that are their strengths, which can better direct their subsequent placement and ultimately job search efforts. With many of our students being offered their first psychologically related job through their WIL experience, the need for students to be forward focused and see their placements as a trajectory of development becomes paramount for employability.

9 Limitations

Throughout the project there was a strong stakeholder focus on the competency evaluation tool delivering reliable and valid information to the student to inform their real-world learning trajectories. Achieving this focus is difficult, as the competency evaluation tool is used by a range of different supervisors from a variety of different organisations, in a multitude of differing contexts. The competency evaluation tool therefore needs to be robust enough to better inform students of their competence but also sufficiently reliable for a range of supervisors to use and provide congruent feedback to students. To this end, the more specific anchors for competence assessment in the new model should assist in enhancing the reliability of competency assessment across supervisors. In addition, the provision of a guide or training for assessors would improve the likelihood of the assessment competency framework being implemented successfully. Further assessments of inter-rater reliability (multiple supervisors assessing the same student on the same placement ) would be ideal, however only a limited number of placements offer the potential for these types of assessments to be undertaken.

10 Benefits

Given the standardisation of the competencies that organisational psychology students acquire while undertaking their WIL experiences, this research is of benefit to other institutions offering organisational psychology training who want to mould their student perceptions of placements as a trajectory of development, rather than a set of independent WIL experiences. Indeed, such tools for assessing WIL competence for individual and successive placements are of benefit in other specialty areas, as well as for professional training post-graduation.

11 Conclusion

WIL is an integral and substantial component of accredited postgraduate psychology training within Australia. With this increasing reliance on the assessment of competence and outputs, an innovative approach was warranted, that combines both formative and summative assessment to ensure students can continue to attain competence while recognising their strengths and areas for development. Through the integration of seven key themes arising from stakeholder feedback, an improved method of assessing competence was developed and trialled. This new competency assessment tool, based on Miller’s pyramid model, provides students with feedback both throughout a placement and across successive placements. The tool thus becomes both an assessment of learning and an assessment for learning (Carless, 2007). Through this improved feedback students can better understand their performance strengths and thereby identify their optimal career paths. This enables students to develop a competency based understanding of performance, so as to assist them as graduates in the workforce, informing their post-practicum professional development.