Abstract
It is shown that if cell weights may be calculated from the data the chance-corrected Zegers-ten Berge coefficients for metric scales are special cases of Cohen’s weighted kappa. The corrected coefficients include Pearson’s product-moment correlation, Spearman’s rank correlation and the intraclass correlation ICC(3, 1).
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
ABRAIRA, V., and PÉREZ DE VARGAS, A. (1999), “Generalization of the Kappa Coefficient for Ordinal Categorical Data, Multiple Observers and Incomplete Designs”, Q ÜESTIIÓ, 23, 561–571.
BERRY, K.J., and MIELKE, P.W. (1988), “A Generalization of Cohen’s Kappa Agreement Measure to Interval Measurement and Multiple Raters”, Educational and Psychological Measurement, 48, 921–933.
BERRY, K.J., JOHNSTON, J.E., and MIELKE, P.W. (2008), “Weighted Kappa for Multiple Raters”, Perceptual and Motor Skills, 107, 837–848.
CICCHETTI, D V. (1976), “Assessing Inter-rater Reliability for Rating Scales: Resolving Some Basic Issues”, British Journal of Psychiatry, 129, 452–456.
CICCHETTI, D.V., and ALLISON, T. (1971), “A New Procedure for Assessing Reliability of Scoring EEG Sleep Recordings”, The American Journal of EEG Technology, 11, 101–110.
CICCHETTI, D., BRONEN, R., SPENCER, S., HAUT, S., BERG, A., OLIVER, P., and TYRER, P. (2006), “Rating Scales, Scales of Measurement, Issues of Reliability. Resolving Some Critical Issues for Clinicians and Researchers”, The Journal of Nervous and Mental Disease, 194, 557–564.
COHEN, J. (1960), “A Coefficient of Agreement for Nominal Scales”, Educational and Psychological Measurement, 20, 37–46.
COHEN, J. (1968), “Weighted Kappa: Nominal Scale Agreement With Provision for Scaled Disagreement or Partial Credit”, Psychological Bulletin, 70, 213–220.
CONGER, A.J. (1980), “Integration and Generalization of Kappas for Multiple Raters”, Psychological Bulletin, 88, 322–328.
CREWSON, P.E. (2005), “Fundamentals of Clinical Research for Radiologists. Reader Agreement Studies”, American Journal of Roentgenology, 184, 1391–1397.
DAVIES, M., and FLEISS, J.L. (1982), “Measuring Agreement for Multinomial Data”, Biometrics, 38, 1047–1051.
FAGOT, R.F. (1993), “A Generalized Family of Coefficients of Relational Agreement for Numerical Scales”, Psychometrika, 58, 357–370.
FLEISS, J.L., and COHEN, J. (1973), “The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability”, Educational and Psychological Measurement, 33, 613–619.
GRAHAM, P., and JACKSON, R. (1993), “The Analysis of Ordinal Agreement Data: Beyond Weighted Kappa”, Journal of Clinical Epidemiology, 46, 1055–1062.
HEUVELMANS, A.P.J.M., and SANDERS, P.F. (1993), “Beoordelaarsovereenstemming”, in Psychometrie in de Praktijk, eds. T.J.H.M. Eggen and P.F. Sanders, Arnhem: Cito Instituut voor Toestontwikkeling, pp. 443–470.
HUBERT, L. (1977), “Kappa Revisited”, Psychological Bulletin, 84, 289–297.
JANSON, H., and OLSSON, U. (2001), “A Measure of Agreement for Interval or Nominal Multivariate Observations”, Educational and Psychological Measurement, 61, 277–289.
JOBSON, J.D. (1976), “A Coefficient of Equality for Questionnaire Items with Interval Scales”, Educational and Psychological Measurement, 36, 271–274.
LIGHT, R.J. (1971), “Measures of Response Agreement for Qualitative Data: Some Generalizations and Alternatives”, Psychological Bulletin, 76, 365–377.
MACLURE, M., and WILLETT, W C. (1987), “Misinterpretation and Misuse of the Kappa Statistic”, Journal of Epidemiology, 126, 161–169.
MCGRAW, K.O., and WONG, S.P. (1996), “Forming Inferences About Some Intraclass Correlation Coefficients”, Psychological Methods, 1, 30–46.
MIELKE, P.W., BERRY, K.J., and JOHNSTON, J.E. (2007), “The Exact Variance of Weighted Kappa With Multiple Raters”, Psychological Reports, 101, 655–660.
MIELKE, P.W., BERRY, K.J., and JOHNSTON, J.E. (2008), “Resampling Probability Values for Weighted Kappa With Multiple Raters”, Psychological Reports, 102, 606–613.
POPPING, R. (1983), “Overeenstemmingsmaten voor Nominale Data”, PhD thesis, Rijksuniversiteit Groningen, Groningen.
POPPING, R. (2010), “Some Views on Agreement to Be Used in Content Analysis Studies”, Quality & Quantity, 44, 1067-1078.
SCHUSTER, C. (2004), “A Note on the Interpretation of Weighted Kappa and Its Relations to Other Rater Agreement Statistics for Metric Scales”, Educational and Psychological Measurement, 64, 243–253.
SCHUSTER, C., and SMITH, D.A. (2005), “Dispersion Weighted Kappa: An Integrative Framework for Metric and Nominal Scale Agreement Coefficients”, Psychometrika, 70, 135-1-46.
SHROUT, P.E., and FLEISS, J.L. (1979), “Intraclass Correlations: Uses in Assessing Rater Reliability”, Psychological Bulletin, 86, 420–428.
STINE, W.W. (1989), “Interobserver Relational Agreement”, Psychological Bulletin, 106, 341–347.
VANBELLE, S., and ALBERT, A. (2009a), “Agreement Between Two Independent Groups of Raters”, Psychometrika, 74, 477–491.
VANBELLE, S., and ALBERT, A. (2009b), “A Note on the Linearly Weighted Kappa Coefficient for Ordinal Scales”, Statistical Methodology, 6, 157–163.
VON EYE, A., and MUN, E.Y. (2006), Analyzing Rater Agreement. Manifest Variable Methods, New Jersey USA: Lawrence Erlbaum Associates.
WARRENS, M.J. (2010), “Inequalities Between Multi-rater Kappas”, Advances in Data Analysis and Classification, 4, 271–286.
WARRENS, M.J. (2011), “Cohen’s Linearly Weighted Kappa Is a Weighted Average of 2 × 2 Kappas”, Psychometrika, 76, 471-486.
WARRENS, M.J. (2012a), “Some Paradoxical Results for the Quadratically Weighted Kappa”, Psychometrika, 77, 315–323.
WARRENS, M.J. (2012b), “A Family of Multi-rater Kappas That Can Always Be Increased and Decreased by Combining Categories”, Statistical Methodology, 9, 330–340.
WARRENS, M.J. (2012c), “Equivalences of Weighted Kappas for Multiple Raters”, Statistical Methodology, 9, 407–422.
WARRENS, M.J. (2013), “Conditional Inequalities Between Cohen’s Kappa and Weighted Kappas”, Statistical Methodology, 10, 14–22.
WINER, B.L. (1971), Statistical Principles in Experimental Design (2nd ed.), New York: McGraw-Hill.
ZEGERS, F.E. (1986a), A General Family of Association Coefficients, Groningen, Netherlands: Boomker.
ZEGERS, F.E. (1986b), “A Family of Chance-corrected Association Coefficients for Metric Scales”, Psychometrika, 51, 559–562.
ZEGERS, F.E. (1991), “Coefficients for Interrater Agreement”, Applied Psychological Measurement, 15, 321–333.
ZEGERS, F.E., and TEN BERGE, J.M.F. (1985), “A Family of Association Coefficients for Metric Scales”, Psychometrika, 50, 17–24.
Author information
Authors and Affiliations
Corresponding author
Additional information
The author thanks three reviewers for their helpful comments and valuable suggestions on a previous version of the manuscript. This research is part of project 451-11-026 funded by the Netherlands Organisation for Scientific Research.
Rights and permissions
About this article
Cite this article
Warrens, M.J. Corrected Zegers-ten Berge Coefficients Are Special Cases of Cohen’s Weighted Kappa. J Classif 31, 179–193 (2014). https://doi.org/10.1007/s00357-014-9156-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00357-014-9156-9