Abstract
This paper extends the theory of conditional covariances to polytomous items. It has been proven that under some mild conditions, commonly assumed in the analysis of response data, the conditional covariance of two items, dichotomously or polytomously scored, given an appropriately chosen composite is positive if, and only if, the two items measure similar constructs besides the composite. The theory provides a theoretical foundation for dimensionality assessment procedures based on conditional covariances or correlations, such as DETECT and DIMTEST, so that the performance of these procedures is theoretically justified when applied to response data with polytomous items. Various estimators of conditional covariances are constructed, and special attention is paid to the case of complex sampling data, such as those from the National Assessment of Educational Progress (NAEP). As such, the new version of DETECT can be applied to response data sets not only with polytomous items but also with missing values, either by design or at random. DETECT is then applied to analyze the dimensional structure of the 2002 NAEP reading samples of grades 4 and 8. The DETECT results show that the substantive test structure based on the purposes for reading is consistent with the statistical dimensional structure for either grade.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Allen, N., Carlson, J.E., & Zelenak, C. (1999). The NAEP 1996 technical report (NCES 1999-452). Washington, DC: Office of Educational Research and Improvement, US Department of Education.
Allen, N., Donoghue, J.R., & Schoeps, T.L. (2001). The NAEP 1998 technical report (NCES 2001-509). Washington, DC: Office of Educational Research and Improvement, US Department of Education.
Anderson, T.W. (1984). An introduction to multivariate statistical analysis (2nd ed.). New York: Wiley.
Beaton, A.E., Johnson, E.G., & Ferris J.J. (1987). The assignment of exercises to students. In A.E. Beaton (Ed.), Implementing the new design: The NAEP 1983–84 technical report (pp. 97–118). Princeton, NJ: Educational Testing Service.
Douglas, J., Kim, H.R., & Stout, W.F. (1994). Exploring and explaining the lack of local independence through conditional covariance functions. Paper presented at the 1994 (April) annual meeting of the American Educational Research Association, New Orleans, LA.
Habing, B., & Roussos, L.A. (2003). On the need for negative local item dependence. Psychometrika, 68, 435–451.
Holland, P.W., & Rosenbaum, P.R. (1986). Conditional association and unidimensionality in monotone latent variable models. The Annals of Statistics, 14, 1523–1543.
Junker, B. (1993). Conditional association, essential independence and monotone unidimensional item response models. The Annals of Statistics, 21, 1359–1378.
Kim, H.R. (1994). New techniques for the dimensionality assessment of standardized test data. Unpublished doctoral dissertation, Department of Statistics, University of Illinois at Urbana-Champaign.
McDonald, R.P. (1994). Testing for approximate dimensionality. In D. Laveault, B. Zumbo, M. Gessaroli, & M. Boss (Eds.), Modern theories of measurement: Problems and issues (pp. 63–85). Ottawa: University of Ottawa Press.
Mislevy, R., & Bock, R.D. (1982). BILOG: Item analysis and test scoring with binary logistic models [Computer software]. Mooresville, IN: Scientific Software.
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.
Muraki, E., & Bock, R.D. (1997). PARSCALE: IRT item analysis and test scoring for rating scale data [Computer software]. Chicago: Scientific Software.
Nandakumar, R., & Stout, W.F. (1993). Refinement of Stout’s procedure for assessing latent trait essential unidimensionality. Journal of Educational Statistics, 18, 41–68.
National Assessment Governing Board. (1992). Reading framework for the National Assessment of Educational Progress: 1992–2002. Washington, DC: National Assessment Governing Board.
National Assessment Governing Board. (2002). Mathematics framework for the 2003 National Assessment of Educational Progress. Washington, DC: National Assessment Governing Board.
Oltman, P.K., Stricker, L.J., & Barrows, T.S. (1990). Analyzing test structure by multidimensional scaling. Journal of Applied Psychology, 75, 21–27.
Reckase, M.D. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401–412.
Reckase, M.D., & McKinley, R.L. (1991). The discriminating power of items that measure more than one dimension. Applied Psychological Measurement, 15, 361–373.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, No. 17. Greensboro, NC: Psychometric Society.
Samejima, F. (1972). A general model for free-response data. Psychometrika Monograph No. 18. Greensboro, NC: Psychometric Society.
Stout, W.F. (1987). A nonparametric approach for assessing latent trait dimensionality. Psychometrika, 52, 589–617.
Stout, W.F., Habing, B., Douglas, J., Kim, H.R., Roussos, L.A., & Zhang, J. (1996). Conditional covariance based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20, 331–354.
Van Abswoude, A.A.H., Van der Ark, L.A., & Sijtsma, K. (2004). A comparative study on test dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28, 3–24.
Wang, M. (1986). Fitting a unidimensional model on the multidimensional item response data (ONR Technical Report 87-1). Iowa City, IA: University of Iowa.
Yang, X., & Zhang, J. (2001). Construction and evaluation of bias-corrected estimators of DETECT dimensionality index. Paper presented at the 2001 (April) annual meeting of the American Educational Research Association, Seattle, WA.
Yu, F., & Nandakumar, R. (2001). Poly-DETECT for quantifying the degree of multidimensionality of item response data. Journal of Educational Measurement, 38, 99–120.
Zhang, J. (1996). Some fundamental issues in item response theory with applications. Unpublished doctoral dissertation, Department of Statistics, University of Illinois at Urbana-Champaign.
Zhang, J., & Stout, W.F. (1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129–152.
Zhang, J., & Stout, W.F. (1999b). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213–249.
Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24, 293–308.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was supported by the Educational Testing Service and the National Assessment of Educational Progress (Grant R902F980001), US Department of Education. The opinions expressed herein are solely those of the author and do not necessarily represent those of the Educational Testing Service. The author would like to thank Ting Lu, Paul Holland, Shelby Haberman, and Feng Yu for their comments and suggestions.
Requests for reprints should be sent to Jinming Zhang, Educational Testing Service, MS 02-T, Rosedale Road, Princeton, NJ 08541, USA. E-mail: jzhang@ets.org
Rights and permissions
About this article
Cite this article
Zhang, J. Conditional Covariance Theory and Detect for Polytomous Items. Psychometrika 72, 69–91 (2007). https://doi.org/10.1007/s11336-004-1257-7
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-004-1257-7