Abstract
Background
To establish credible, defensible and acceptable passing scores for written tests is a challenge for health profession educators. Angoff procedures are often used to establish pass/fail decisions for written and performance tests. In an Angoff procedure judges’ expertise and professional skills are assumed to influence their ratings of the items during standard-setting. The purpose of this study was to investigate the impact of judges’ item-related knowledge on their judgement of the difficulty of items, and second, to determine the stability of differences between judges.
Method
Thirteen judges were presented with two sets of 60 items on different occasions. They were asked to not only judge the difficulty of the items but also to answer them, without the benefit of the answer key. For each of the 120 items an Angoff estimate and an item score were obtained. The relationship between the Angoff estimate and the item score was examined by applying a regression analysis to the 60 items (Angoff estimate, score) for each judge at each occasion.
Results and conclusions
This study shows that in standard-setting the individual judgement of the individual item is not only a reflection of the difficulty of the item but also of the inherent stringency of the judge and his/her subject-related knowledge. Considerable variation between judges in their stringency was found, and Angoff estimates were significantly affected by a judge knowing or not knowing the answer to the item. These findings stress the importance of a careful selection process of the Angoff judges when making pass/fail decisions in health professions education. They imply that judges should be selected who are not only capable of conceptualising the ‘minimally competent student’, but who would also be capable of answering all the items.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Angoff W.H. (1971) Scales, norms and equivalent scores. In: Thorndike R.L. (eds) Educational Measurement. American Council on Education, Washington DC, pp 508–600
Chang L., Dziuban C.D., Hynes M.C. (1996) Does a standard reflect minimal competency of examines or judge competency? Applied Measurement in Education 9(2):161–173
Cusimano M.D. (1996) Standard-setting in medical education. Academic Medicine 71(10 Suppl):112–120
Cusimano M.D., Rothman A.I. (2003) The effect of incorporating normative data into a criterion-referenced standard setting in medical education. Academic Medicine 78(10 Suppl):88–90
Downing S.M., Tekian A., Yudkowsky R. (2006) Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teaching and Learning in Medicine 18(1):50–57
Norcini J.J., Lipner R.S., Langdon L.O., Strecker C.A. (1987) A comparison of three variations on a standard-setting method. Journal of Educational Measurement 24(1):56–64
Norcini J.J., Shea J.A., Ping J.C. (1988). A note on the application of multiple matrix sampling to standard-setting. Journal of Educational Measurement 25(2):159–164
Norcini J.J., Shea J.A. (1992) The reproducibility of standards over groups and occasions. Applied Measurement in Education 5: 63–72
Norcini J.J., Shea J.A. (1997) The credibility and comparibility of standards. Applied Measurement in Education 10:39–59
Plake, B.S., Melican, G.J. & Mills, C.N. (1991). Factors influencing intrajudge consistency during standard-setting. Educational Measurement Issues and practice 10(2): 15–16, 22, 25–26
Saunders J.C., Ryan J.P., Huynh H. (1981) A comparison of two approaches to setting passing scores based on the Nedelsky procedure. Applied Psychological Measurement 5:209–217
Verhoeven B.H., Verwijnen G.M., Muijtjens A.M.M., Scherpbier A.J.J.A., Van der Vleuten C.P.M. (2002) Panel expertise for an Angoff standard-setting procedure in progress testing. Item writers compared to recently graduated students. Medical Education 36(9):860–867
Williams R.G., Klamen D.A., McGaghie W.C. (2003) Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine 15(4):270–292
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Verheggen, M.M., Muijtjens, A.M.M., Van Os, J. et al. Is an Angoff Standard an Indication of Minimal Competence of Examinees or of Judges?. Adv in Health Sci Educ 13, 203–211 (2008). https://doi.org/10.1007/s10459-006-9035-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-006-9035-1