Abstract
Nonparametric item response models have been developed as alternatives to the relatively inflexible parametric item response models. An open question is whether it is possible and practical to administer computerized adaptive testing with nonparametric models. This paper explores the possibility of computerized adaptive testing when using nonparametric item response models. A central issue is that the derivatives of item characteristic Curves may not be estimated well, which eliminates the availability of the standard maximum Fisher information criterion. As alternatives, procedures based on Shannon entropy and Kullback–Leibler information are proposed. For a long test, these procedures, which do not require the derivatives of the item characteristic eurves, become equivalent to the maximum Fisher information criterion. A simulation study is conducted to study the behavior of these two procedures, compared with random item selection. The study shows that the procedures based on Shannon entropy and Kullback–Leibler information perform similarly in terms of root mean square error, and perform much better than random item selection. The study also shows that item exposure rates need to be addressed for these methods to be practical.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B.N. Petrov, & F. Csaki, (Eds.), Second international symposium on information theory (pp. 267–281); Budapest: Akadémiai Kiadó.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F.M. Lord, & M.R. Novick (Eds.), Statistical theories of mental test scores (pp. 397–479) Reading, MA: Addison-Wesley.
Chang, H.H.,& Stout, W.F. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37–52.
Chang, H.H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229.
Cover, T.M., & Thomas, J.A. (1991). Elements of information theory. New York: Wiley.
DeGroot M.H. (1962). Uncertainty, information and sequential experiments. Annals of Mathematical Statistics, 33, 404–419.
Douglas, J. (1997). Joint consistency of nonparametric item characteristic curve and ability estimation. Psychometrika, 62, 7–28.
Eubank, R.L. (1988). Spline smoothing and nonparametric regression. New York, Marcel Dekker.
Grayson, D.A. (1988). Two-group classification in latent trait theory: Scores with monotone likelihood ratio. Psychometrika, 53, 383–392.
He, X., & Ng, P. (1998). COBS: Qualitatively constrained smoothing via linear programming. Unpublished manual for SCOBS.
Nadaraya, E.A. (1964). On estimating regression. Probability Theory and its Applications, 9, 141–142.
Ramsay, J.O. (1991). Kernel smoothing approaches to nonparametric item characteristic curve estimation. Psychometrika, 56, 611–630.
Ramsay, J.O. (2000). TESTGRAF: A program for the graphical analysis of multiple choice test and questionnaire data [Computer Program]. Montreal: McGill University.
Ramsay, J.O., & Abrahamowicz, M. (1989). Binomial regression with monotone splines: A psychometric application. Journal of the American Statistical Association, 84, 906–915.
Ramsay, J.O., & Winsberg, S. (1991). Maximum marginal likelihood estimation for semiparametric item analysis. Psychometrika, 56, 365–379.
Rossi, N., Wang, X., & Ramsay, J.O. (2002). Nonparametric item response function estimates with the EM algorithm. Journal of Educational and Behavioral Statistics, 27, 291–317.
Shannon, C.E. (1948). A mathematical theory of communication, Bell Systems Techical Journal, 27, 379–423, 623–656.
Tatsuoka, C. (2002). Data analytic methods for latent parially ordered classification models. Journal of the Royal Statistical Society, Series C, 51, 337–350.
Tatsuoka, C., & Ferguson, T. (2003). Sequential classification on patially ordered sets. Journal of Royal Statistical Society, Series B, 65, 143–158.
van der Linden, W.J., & Glas, C.A.W. (2000). Computerized adaptive testing: Theory and practice. Dordrecht: Kluwer Academic.
Walker, A.M. (1969). On the asymptotic behavior of posterior distributions. Journal of the Royal Statistical Society, Series B, 31, 80–88.
Watson, G.S. (1964). Smooth regression analysis. Sankhya, Series A, 26, 359–372.
Xu X., Chang, H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Presented at the Annual Meeting of the National Council of Measurement in Education, Chicago, April 2003.
Author information
Authors and Affiliations
Additional information
The authors would like to thank Hua Chang for his help in conducting this research.
Rights and permissions
About this article
Cite this article
Xu, X., Douglas, J. Computerized adaptive testing under nonparametric IRT models. Psychometrika 71, 121–137 (2006). https://doi.org/10.1007/s11336-003-1154-5
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-003-1154-5