Abstract
In the process of coding open-ended questions, the evaluation of interjudge reliability is a critical issue. In this paper, using real data, the behavior of three coefficients of reliability among coders, Cohen’s K, Krippendorff’s α and Perreault and Leigh’s I r are patterned, in terms of the number of judges involved and the categories of answer defined. The outcome underlines the importance of both variables in the valuations of interjudge reliability, as well as the higher adequacy of Perreault and Leigh’s I r and Krippendorff’s α for marketing and opinion research.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
A.M. Abernethy G.R. Franke (1996) ArticleTitleThe information content of advertising: A meta-analysis Journal of Advertising 25 IssueID2 1–17
Bardin, L. (1977). L’analyse de contenu. (Paris: PUF).
E.M. Bennett A.C. Goldstein (1954) ArticleTitleCommunications through limited response questioning Public Opinion Quaterly 18 303–308 Occurrence Handle10.1086/266520
Berelson, B. (1952). Content Analysis in Communications Research. Glencoe (Ill.): Free Press.
H. Brenner U. Kliebsch (1996) ArticleTitleDependence of weighted kappa coefficients on the number of categories Epidemiology 7 IssueID2 199–202
J. Cohen (1960) ArticleTitleA coefficient of agreement for nominal scales Educational and Psychological Measurement 20 IssueIDWinter 37–46
N.K. Denzin Y.S. Lincoln (1994) Handbook of Qualitative Research Sage Publications Thousand Oaks (CA)
ESOMAR (2003). Annual Study of the Market Research Industry 2002. (Amsterdam: ESOMAR).
J.L. Fleiss (1971) ArticleTitleMeasuring nominal scale agreement among many raters Psychological Bulletin 76 378–382 Occurrence Handle10.1037/h0031619
A. Fontana J.H. Frey (1994) Interviewing. The art of science N.K. Denzin Y.S. Licoln (Eds) Handbook of Qualitative Research Sage Publications Thousand Oaks (CA)
B.G. Glaser A.L. Strauss (1999) The Discovery of Grounded Theory: Strategies for Qualitative Research Aldine de Gruyter New York
K. Grayson R. Rust (2001) ArticleTitleInterrater reliability Journal of Consumer Psychology 10 IssueID1&2 71–73 Occurrence Handle10.1207/S15327663JCP1001&2_06
L. Herbert (1977) ArticleTitleKappa revisited Psychological Bulletin 84 IssueID2 289–297 Occurrence Handle10.1037/0033-2909.84.2.289
M.B. Holbrook (1977) ArticleTitleMore on content analysis in consumer research Journal of Consumer Research 4 IssueID3 176–177 Occurrence Handle10.1086/208694
O.R. Holsti (1969) Content Analysis for the Social Sciences and Humanities Addison-Wesley Reading (MA)
Hsu Fied (2003) ArticleTitleInterrater agreement measures: Comments on Kappa, Cohen’s Kappa, Scott’s π and Aicking α Understanding statistics 2 IssueID3 205–219 Occurrence Handle10.1207/S15328031US0203_03
M.A. Hughes D.E. Garret (1990) ArticleTitleIntercoder reliability estimation approaches in marketing: A Generability Theory framework for quantitative data Journal of Marketing Research 27 IssueID2 185–95 Occurrence Handle10.2307/3172845
N. Kang A. Kara H.A. Laskey F.B. Seaton (1993) ArticleTitleA SAS MACRO for calculating intercoder agreement in content analysis Journal of Advertising 22 IssueID2 17–28
H.H. Kassarjian (1977) ArticleTitleContent analysis in consumer research Journal of Consumer Research 4 IssueID2 8–18 Occurrence Handle10.1086/208674
R.H. Kolbe M.S. Burnett (1991) ArticleTitleContent-analysis research: An examination of applications with directives for improving research reliability and objectivity Journal of Consumer Research 18 IssueIDSeptember 243–250 Occurrence Handle10.1086/209256
K. Krippendorff (1971) ArticleTitleReliability of recording instructions: Multivariate agreement for nominal data Behavioral Science 16 IssueID3 228–235
K. Krippendorff (1980) Content Analysis, an Introduction to Its Methodology Sage Publications Thousand Oaks (CA)
J.R. Landis G.G. Koch (1977) ArticleTitleThe measurement of observer agreement for categorical data Biometrics 33 159–174 Occurrence Handle10.2307/2529310
R.L. Lehmann S. Gupta J. H. Steckel (1998) Marketing Research Addison-Wesley Educational Publishers Inc. Reading (MA)
R.J. Light (1971) ArticleTitleMeasures of response agreement for qualitative data: some generalizations and alternatives Psychological Bulleting 76 IssueID5 365–377 Occurrence Handle10.1037/h0031643
Y.S. Lincoln E.G. Guba (1985) Naturalistic Inquiry Sage Publications Beverly Hills (CA)
M. Lombard J. ZINDER-duch C. Campanela (2002) ArticleTitleContent analysis in mass communication: Assessment and reporting of intercoder reliability Human Communication Research 28 IssueID4 587–604 Occurrence Handle10.1093/hcr/28.4.587
López-Aranguren (1989). El análisis de contenido. In: M. García et al. (Coord.): El Análisis de la Realidad Social: Métodos y Técnicas de Investigación. Madrid: Alianza Editorial.
T. Luque (1997) Investigación de Marketing Ed. Ariel Barcelona
K. Mackensen U. Wille (1999) ArticleTitleQualitative text analysis supported by conceptual data systems Quality & Quantity 33 135–156 Occurrence Handle10.1023/A:1004305723553
M.B. Miles A.M. Huberman (1994) Quality Data Analysis An expanded Sourcebook. Sage Publications Thousand Oaks (CA)
W.L. Miller B.F. Crabtree (1994) Clinical research N.K. Denzin Y.S. Licoln (Eds) Handbook of Qualitative Research Sage Publications Thousand Oaks (CA)
W.D. Perreault E.L. LEIGH (1989) ArticleTitleReliability of nominal data based on quantitative judgments Journal of Marketing Research 23 IssueIDMay 130–43
Richards Richards (1994) Using computers in qualitative research N.K. Denzin Y.S. Licoln (Eds) Handbook of Qualitative Research Sage Publications Thousand Oaks (CA)
D. Riffe A.A. Freitag (1997) ArticleTitleA content analysis of content analysis: Twenty-five years of Journalism Quarterly Journalism & Mass Communication Quarterly 74 873–882
C.W. Roberts (2000) ArticleTitleA conceptual framework for quantitative text analysis: On joining probabilities and substantive inferences about text Quality & Quantity 34 259–274 Occurrence Handle10.1023/A:1004780007748
R.T. Rust B. Cooil (1994) ArticleTitleReliability measures for qualitative data: Theory and implications Journal of Marketing Research 31 1–14 Occurrence Handle10.2307/3151942
S. Spiggle (1994) ArticleTitleAnalysis and interpretation of qualitative data in consumer research Journal of Marketing Research 21 IssueIDDecember 491–503
E.L. Spitznagel J.E. Helzer (1985) ArticleTitleA proposed solution to the base rate problem in the kappa statistic Archives of General Psychiatry 42 IssueID7 725–28
A. Strauss J. Corbin (1990) Basic of qualitative research: Grounded theory procedures and techniques Sage Publications Newbury Park (CA)
R.P. Weber (1985) Basic Content Analysis Ed. Sage Publications Newbury Park
E.A. Weitzman M.B. Miles (1995) Computer Programs for Qualitative Data Analysis: A Software Sourcebook Sage Publications Thousand Oaks (CA)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Leiva, F.M., Ríos, F.J.M. & Martínez, T.L. Assessment of Interjudge Reliability in the Open-Ended Questions Coding Process. Qual Quant 40, 519–537 (2006). https://doi.org/10.1007/s11135-005-1093-6
Issue Date:
DOI: https://doi.org/10.1007/s11135-005-1093-6