Abstract
Maximum likelihood and Bayesian procedures for item selection and scoring of multidimensional adaptive tests are presented. A demonstration using simulated response data illustrates that multidimensional adaptive testing (MAT) can provide equal or higher reliabilities with about one-third fewer items than are required by one-dimensional adaptive testing (OAT). Furthermore, holding test-length constant across the MAT and OAT approaches, substantial improvements in reliability can be obtained from multidimensional assessment. A number of issues relating to the operational use of multidimensional adaptive testing are discussed.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Ackerman, T. A. (1989). Unidimensional IRT calibration of compensatory and non-compensatory multidimensional items.Applied Psychological Measurement, 13, 113–127.
Ackerman, T. A. (1991). The use of unidimensional parameter estimates of multidimensional items in adaptive testing.Applied Psychological Measurement, 15, 13–24.
Anderson, T. W. (1984).An introduction to multivariate statistical analysis (2nd ed.). New York: John Wiley & Sons.
Ansley, T. N., & Forsyth, R. A. (1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from two-dimensional data.Applied Psychological Measurement, 9, 37–48.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.),Statistical theories of mental test scores (pp. 397–479). Reading, MA: Addison-Wesley.
Bloxom, B., & Vale, C. D. (1987, June).Multidimensional adaptive testing: An approximate procedure for updating. Paper presented at the meeting of the Psychometric Society, Montreal.
Bock, R. D., Gibbons, R., & Muraki, E. (1988). Full-information item factor analysis.Applied Psychological Measurement, 12, 261–280.
Bollen, K. A. (1989).Structural equations with latent variables. New York: John Wiley & Sons.
Carlson, J. E. (1987).Multidimensional item response theory estimation: A computer program (Research Report ONR 87-2). Iowa City, IA: The American College Testing Program.
Drasgow, F., Levine, M. V., & McLaughlin, M. E. (1991). Appropriateness measurement for some multidimensional test batteries.Applied Psychological Measurement, 15, 171–191.
Drasgow, F., & Parsons, C. K. (1983). Application of unidimensional item response theory models to multidimensional data.Applied Psychological Measurement, 7, 189–199.
Folk, V. G., & Green, B. F. (1989). Adaptive estimation when the unidimensionality assumption of IRT is violated.Applied Psychological Measurement, 13, 373–389.
Fraser, C. (1988).NOHARM II. A Fortran program for fitting unidimensional and multidimensional normal ogive models of latent trait theory. Armidale, Australia: The University of New England, Center for Behavioral Studies.
Harrison, D. A. (1986). Robustness of IRT parameter estimation to violations of the unidimensionality assumption.Journal of Educational Statistics, 11, 91–115.
Hattie, J. (1981).Decision criteria for determining unidimensionality. Unpublished doctoral dissertation, University of Toronto, Canada.
Hendrickson, A. E., & White, P. O. (1964). PROMAX: A quick method for rotation to oblique simple structure.British Journal of Mathematical and Statistical Psychology, 17, 65–70.
IMSL (1991).International Mathematical and Statistical Libraries (Stat/Library), User's Manual, Houston, TX: Author.
Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis.Psychometrika, 23, 187–200.
Lord, F. M. (1980).Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
McDonald, R. P. (1985). Unidimensional and multidimensional models for item response theory. In D. J. Weiss (Ed.),Proceedings of the 1982 Computerized Adaptive Testing Conference (pp. 127–148). Minneapolis: University of Minnesota, Department of Psychology, Psychometrics Methods Program.
McKinley, R. L. (1989).Confirmatory analysis of test structure using multidimensional item response theory (Report No. RR-89-31). Princeton, NJ: Educational Testing Service.
McKinley, R. L., & Reckase, M. D. (1983). MAXLOG: A computer program for the estimation of the parameters of a multidimensional logistic model.Behavior Research Methods & Instrumentation, 15, 389–390.
Miller, T., Reckase, M. D., Spray, J. A., Luecht, R., & Davey, T. (in press).Multidimensional Item Response Theory.
Mislevy, R. J. (1984). Estimating latent distributions.Psychometrika, 49, 359–381.
Mislevy, R. J. (1989).PC-BILOG: Item analysis and test scoring with binary logistic models. Mooresville, IN: Scientific Software.
Moreno, K. E., & Segall, D. O. (1993). CAT-ASVAB Precision.Proceedings of the 34th Annual Conference of the Military Testing Association. San Diego: Navy Personnel Research and Development Center.
Muthén, B. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators.Psychometrika, 49, 115–132.
Owen, R. J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing.Journal of the American Statistical Association, 70, 351–356.
Prestwood, J. S., Vale, C. D., Massey, R. H., & Welsh, J. R. (1985).Armed Services Vocational Aptitude Battery: Development of an adaptive item pool (Technical Report 85-19). San Antonio, TX: Air Force Human Resources Laboratory.
Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications.Journal of Educational Statistics, 4, 207–230.
Reckase, M. D., Ackerman, T. A., & Carlson, J. E. (1988). Building a unidimensional test using multidimensional items.Journal of Educational Measurement, 25, 193–203.
Searle, S. R. (1982).Matrix algebra useful for statistics. New York: John Wiley & Sons.
Segall, D. O., Moreno, K. E., & Hetter, R. D. (1987).ACAP item pools: Analysis and recommendations. Unpublished manuscript, Navy Personnel Research and Development Center, San Diego.
Sympson, J. B., & Hetter, R. D. (1985, October).Controlling item exposure rates in computerized adaptive tests. Paper presented at the 27th Annual meeting of the Military Testing Association, San Diego, CA.
Tam, S. S. (1992).A comparison of methods for adaptive estimation of a multidimensional trait. Unpublished doctoral dissertation, Columbia University.
Wainer, H. W., Dorans, N. J., Flaugher, R., Green, B. F., Mislevy, R. J., Steinberg, L., & Thissen, D. (1990).Computerized adaptive testing: A primer. Hillsdale, NJ: Erlbaum.
Way, W. D., Ansley, T. N., & Forsyth, R. A. (1988). The comparative effects of compensatory and noncompensatory two-dimensional data on unidimensional IRT estimation.Applied Psychological Measurement, 12, 239–252.
Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model.Applied Psychological Measurement, 8, 125–145.
Author information
Authors and Affiliations
Additional information
The work reported in this paper was sponsored by the Office of Naval Research. The author wishes to thank the three anonymous reviewers for their useful comments on an earlier version of this manuscript. The opinions expressed in this article are those of the Author, are not official and do not necessarily reflect the views of the Navy Department.
Rights and permissions
About this article
Cite this article
Segall, D.O. Multidimensional adaptive testing. Psychometrika 61, 331–354 (1996). https://doi.org/10.1007/BF02294343
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF02294343