Abstract
It is shown that approaches other than the internal consistency method of estimating test reliability are either less satisfactory or lead to the same general results. The commonly attendant assumption of a single factor throughout the test items is challenged, however. The consideration of a test made up ofK sub-tests each composed of a different orthogonal factor disclosed that the assumption of a single factor produced an erroneous estimate of reliability with a ratio of (n−K)/(n−1) to the correct estimate. Special difficulties arising from this error in application of current techniques to short tests or to test batteries are discussed. Application of this same multi-factor concept to item-analysis discloses similar difficulties in that field. The item-test coefficient approaches √1/K as an upper limit rather than 1.00 and approaches √1/n as a lower limit rather than .00. This latter finding accounts for an over-estimation error in the Kuder-Richardson formula (8). A new method of isolating sub-tests based upon the item-test coefficient is proposed and tentatively outlined. Either this new method or a complete factor analysis is regarded as the only proper approach to the problem of test reliability, and the item-sub-est coefficient is similarly recommended as the proper approach for item analysis.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Brown, W. Some experimental results in the correlation of mental abilities.Brit. J. Psychol., 1909–1910,3, 296–322.
Edgerton, H. A., and Kolbe, L. E. The method of minimum variation for the combination of criteria.Psychometrika, 1936,1, 183–187.
Edgerton, H. A., and Thomson, K. F. Test scores examined with the lexis ratio.Psychometrika, 1942,7, 281–288.
Horst, P. Obtaining a composite measure from a number of different measures of the same attribute.Psychometrika, 1936,1, 53–60.
Hotelling, H. The most predictable criterion.J. educ. Psychol., 1935,26, 139–142.
Hoyt, C. Test reliability estimated by analysis of variance.Psychometrika, 1941,6, 153–160.
Jackson, R. W. B. Reliability of mental tests.Brit. J. Psychol., 1938–39,29, 267–287.
Kelley, T. L. The reliability coefficient.Psychometrika, 1942,7, 75–83.
Kuder, G. F., and Richardson, M. W. The theory of the estimation of reliability.Psychometrika, 1937,2, 151–160.
Lentz, T. F., and Whitmer, E. F. Item synonymization: a method for determining the total meaning of pencil-paper reactions.Psychometrika, 1941,6, 131–139.
Mosier, C. A note on item analysis and the criterion of internal consistency,Psychometrika, 1936,1, 275–282.
Paulsen, G. B. A coefficient of trait variability.Psychol. Bull., 1931,28, 218.
Richardson, M. W. Note on the rationale of item analysis,Psychometrika, 1936,1, 69–76.
Royer, E. B. Some recent developments in test construction.Proc. Okla. Acad. Sci., 1936,16, 107–109.
Spearman, C. Correlation calculated from faulty data.Brit. J. Psychol., 1909–1910,3, 271–295.
Thompson, G. A. Weighting for battery reliability and prediction.Brit. J. Psychol., 1939–40,30, 357–366.
Thouless, R. H. Test unreliability and functional fluctuation.Brit. J. Psychol., 1935–36,26, 325–343.
Woodrow, H. Quotidian variability.Psychol. Rev., 1932,32, 245–256.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Wherry, R.J., Gaylord, R.H. The concept of test and item reliability in relation to factor pattern. Psychometrika 8, 247–264 (1943). https://doi.org/10.1007/BF02288707
Issue Date:
DOI: https://doi.org/10.1007/BF02288707