Abstract
In this first chapter, we set the stage for subsequent chapters that we believe will push the boundaries of our current thinking about response processes as validity evidence. Evidence based on response processes has been an overlooked source of validity evidence, but one that offers much promise and strength to support the inferences we make from test scores and to provide explanations for test score variance. In this chapter, we address the critical, but often overlooked step of defining response processes. We next take a brief look back at some key moments and players in the history of response processes theorizing and research. We also examine the prevalence of validity evidence based on response processes reported in the published research literature. We conclude by setting a course for the future.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Henceforth referred to as the Standards.
- 2.
Messick (1989a, 1990) would agree with this view but noted that the dominant view of content validation focuses on expert judgments about test content representativeness and relevance. It is because the dominant view of content validity does not address response consistencies and test scores that Messick (1989b) argued that “so-called content validity does not qualify as validity at all” (p. 7).
References
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME]. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME]. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
American Psychological Association, American Educational Research Association, & National Council on Measurement in Education [APA, AERA, & NCME]. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
Ark, T. K., Ark, N., & Zumbo, B. D. (2014). Validation practices of the Objective Structured Clinical Examination (OSCE). In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 267–288). New York, NY: Springer.
Beckman, T. J., Cook, D. A., & Mandrekar, J. N. (2005). What is the validity evidence for assessments of clinical teaching? Journal of General Internal Medicine, 20, 1159–1164. doi:10.1111/j.1525-1497.2005.0258.x.
Brückner, S., & Pellegrino, J. W. (2016). Integrating the analysis of mental operations into multilevel models to validate an assessment of higher education students’ competency in business and economics. Journal of Educational Measurement, 53, 293–312. doi:10.1111/jedm.12113.
Chan, E. K. H. (2014). Standards and guidelines for validation practices: Development and evaluation of measurement instruments. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 9–24). New York, NY: Springer.
Chan, E. K. H., Munro, D. W., Huang, A. H. S., Zumbo, B. D., Vojdanijahromi, R., & Ark, N. (2014). Validation practices in counseling: Major journals, mattering instruments, and the Kuder Occupational Interest Survey (KOIS). In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 67–87). New York, NY: Springer.
Chan, E. K. H., Zumbo, B. D., Chen, M. Y., Zhang, W., Darmawanti, I., & Mulyana, O. P. (2014). Reporting of measurement validity in articles published in Quality of Life Research. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 217–228). New York, NY: Springer.
Chan, E. K. H., Zumbo, B. D., Darmawanti, I., & Mulyana, O. P. (2014). Reporting of validity evidence in the field of health care: A focus on papers published in Value in Health. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 257–265). New York, NY: Springer.
Chan, E. K. H., Zumbo, B. D., Zhang, W., Chen, M. Y., Darmawanti, I., & Mulyana, O. P. (2014). Medical Outcomes Study Short Form-36 (SF-36) and the World Health Organization Quality of Life (WHOQoL) assessment: Reporting of psychometric validity evidence. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 243–255). New York, NY: Springer.
Chinni, M., & Hubley, A. M. (2014). The Satisfaction with Life Scale (SWLS): A review of reported validation practice. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 35–66). New York, NY: Springer.
Cizek, G. J., Rosenberg, S. L., & Koons, H. H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68, 397–412. doi:10.1177/0013164407310130.
Collie, R. J., & Zumbo, B. D. (2014). Validity evidence in the Journal of Educational Psychology: Documenting current practice and a comparison with earlier practice. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 113–135). New York, NY: Springer.
Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. American Journal of Medicine, 119, 166.e7–166.e16. http://doi.org/10.1016/j.amjmed.2005.10.036.
Cook, D. A., Zendejas, B., Hamstra, S. J., Hatala, R., & Brydges, R. (2014). What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Advances in Health Sciences Education, 19, 233–250. doi:10.1007/s10459-013-9458-4.
Cox, D. W., & Owen, J. J. (2014). Validity evidence for a perceived social support measure in a population health context. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 229–241). New York, NY: Springer.
Downing, S. M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37, 830–837. doi:10.1046/j.1365-2923.2003.01594.x.
Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186. doi:10.1007/BF02294171.
Embretson, S. (1993). Psychometric models for learning and cognitive processes. In N. Frederiksen, R. J. Mislevy, & I. I. Bejar (Eds.), Test theory for a new generation of tests (pp. 125–150). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Embretson, S., Schneider, L. M., & Roth, D. L. (1986). Multiple processing strategies and the construct validity of verbal reasoning tests. Journal of Educational Measurement, 23, 13–32. doi:10.1111/j.1745-3984.1986.tb00231.x.
Embretson, S. E. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179–197.
Embretson, S. E., & Schneider, L. M. (1989). Cognitive component models for psychometric analogies: Conceptually driven versus interactive process models. Learning and Individual Differences, 1, 155–178. doi:10.1016/1041-6080(89)90001-0.
Embretson, S. E., & Yang, X. (2013). A multicomponent latent trait model for diagnosis. Psychometrika, 78, 14–36. doi:10.1007/s11336-012-9296-y.
Gorin, J. S., & Embretson, S. E. (2006). Item difficulty modeling of paragraph comprehension items. Applied Psychological Measurement, 30, 394–411. doi:10.1177/0146621606288554.
Gunnell, K. E., Schellenberg, B. J. I., Wilson, P. M., Crocker, P. R. E., Mack, D. E., & Zumbo, B. D. (2014). A review of validity evidence presented in the Journal of Sport and Exercise Psychology (2002–2012): Misconceptions and recommendations for validation research. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 137–156). New York, NY: Springer.
Gunnell, K. E., Wilson, P. M., Zumbo, B. D., Crocker, P. R. E., Mack, D. E., & Schellenberg, B. J. I. (2014). Validity theory and validity evidence for scores derived from the Behavioural Regulation in Exercise Questionnaire. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 175–191). New York, NY: Springer.
Hubley, A. M., Zhu, M., Sasaki, A., & Gadermann, A. (2014). A synthesis of validation practices in the journals Psychological Assessment and European Journal of Psychological Assessment. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 193–213). New York, NY: Springer.
Ivie, J. L., & Embretson, S. E. (2010). Cognitive process modeling of spatial ability: The assembling objects task. Intelligence, 38, 324–335. doi:10.1016/j.intell.2010.02.002.
Lennon, R. T. (1956). Assumptions underlying the use of content validity. Educational and Psychological Measurement, 16, 294–304.
Lyons-Thomas, J., Liu, Y., & Zumbo, B. D. (2014). Validation practices in the social, behavioral, and health sciences: A synthesis of syntheses. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (p. 313319). New York, NY: Springer.
McBride, H. L., Wiens, R. M., McDonald, M. J., Cox, D. W., & Chan, E. K. H. (2014). The Edinburgh Postnatal Depression Scale (EPDS): A review of the reported validity evidence. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 157–174). New York, NY: Springer.
Messick, S. (1972). Beyond structure: In search of functional models of psychological process. Psychometrika, 37, 357–375.
Messick, S. (1989a). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18, 5–11.
Messick, S. (1989b). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY: Macmillan Publishing Co, Inc.
Messick, S. (1990). Validity of test interpretation and use. Princeton, NJ: Educational Testing Service.
Messick, S. (1995). Validity of psychological assessment. Validation of inferences from persons’ responses and performance as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
Mislevy, R. J. (2009). Validity from the perspective of model-based reasoning. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions, and applications (pp. 83–108). Charlotte, NC: Information Age.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). On the role of task model variables in assessment design. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp. 97–128). Mahwah, NJ: Lawrence Erlbaum.
Padilla, J.-L., & Benítez, I. (2014). Validity evidence based on response processes. Psicothema, 26, 136–144.
Pellegrino, J. W., Baxter, G. P., & Glaser, R. (1999). Addressing the “two disciplines” problem: Linking theories of cognition and learning with assessment and instructional practice. Review of Research in Education, 24, 307–353.
Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A framework for conceptualizing and evaluating the validity of instructionally relevant assessments. Educational Psychologist, 51, 59–81. doi:10.1080/00461520.2016.1145550.
Pellegrino, J. W., & Glaser, R. (1979). Cognitive correlates and components in the analysis of individual differences. Intelligence, 3, 187–215.
Sandilands, D., & Zumbo, B. D. (2014). (Mis)alignment of medical education validation research with contemporary validity theory: The Mini-CEX as an example. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 289–310). New York, NY: Springer.
Shear, B. R., & Zumbo, B. D. (2014). What counts as evidence: A review of validity studies in Educational and Psychological Measurement. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 91–111). New York, NY: Springer.
Tourangeau, R. (1984). Cognitive science and survey methods: A cognitive perspective. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey design: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.
Villalobos Coronel, M. (2015). Synthesis of reliability and validation practices used with the Rosenberg self-esteem scale. Master’s thesis, University of British Columbia. Retrieved from https://open.library.ubc.ca/cIRcle/collections/24/items/1.0165784
Whitely (Embretson), S. E. (1977). Information-processing on intelligence test items: Some response components. Applied Psychological Measurement, 1, 465–476. doi:10.1177/014662167700100402.
Whitely (Embretson), S. E. (1980). Multicomponent latent trait models for ability tests. Psychometrika, 45, 479–494.
Zumbo, B. D. (2014). What role does, and should, the test Standards play outside of the United States of America? Educational Measurement: Issues and Practice, 33, 31–33.
Zumbo, B. D., & Chan, E. K. H. (Eds.). (2014). Validity and validation in social, behavioral, and health sciences. New York, NY: Springer.
Zumbo, B. D., Chan, E. K. H., Chen, M. Y., Zhang, W., Darmawanti, I., & Mulyana, O. P. (2014). Reporting of measurement validity in articles published in Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement. In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 27–34). New York, NY: Springer.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Hubley, A.M., Zumbo, B.D. (2017). Response Processes in the Context of Validity: Setting the Stage. In: Zumbo, B., Hubley, A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-56129-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-56129-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-56128-8
Online ISBN: 978-3-319-56129-5
eBook Packages: Social SciencesSocial Sciences (R0)