Abstract
The following study investigates the measurement equivalence of an online and paper-and-pencil (PAP) survey of human values. For this purpose, a total of 250 respondents completed the 21-item version of the Portrait Value Questionnaire either online (n = 125) or by PAP (n = 125). This questionnaire was developed by Schwartz (Advances in experimental social psychology, Academic Press, New York, 1992) and has been included in the European Social Survey since 2002 to test his theory of basic human values. Measurement invariance was tested via a multiple group confirmatory factor analysis. The assessment of invariance included the three levels of configural, metric, and scalar invariance, and the latent means of the values between both samples were compared. Results of this study show that the measurements are invariant at the three levels (configural, metric, and scalar), but there are latent mean differences between the values across the surveys. These differences may be partly explained by age and level of education differences between the two samples. Based on these findings we conclude that the methods of measurement are essentially invariant.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Arbuckle J.L.: Amos 6.0 User’s Guide. SPSS, Chicago (2005)
Bachmann D.P., Elfrink J., Vazzana G.: E-mail and snail mail face off in rematch. Mark. Res. 11(4), 11–15 (1999)
Bandilla W.: Web surveys—an appropriate mode of data collection for the social sciences?. In: Batinic, B., Reips, U.-D., Bosnjak, M. (eds) Online Social Sciences, pp. 1–8. Hogrefe & Huber Publishers, Seattle (2002)
Bandilla W., Bosnjak M., Altdorfer P.: Effekte des Erhebungsverfahrens? Ein Vergleich zwischen einer web-basierten und einer schriftlichen Befragung zum ISSP Modul Umwelt. ZUMA-Nachrichten 49, 7–28 (2001)
Batinic B.: Fragebogenuntersuchungen im Internet. Shaker, Aachen (2001)
Batinic B., Bosnjak M.: Fragebogenuntersuchungen im Internet. In: Batinic, B. (eds) Internet für Psychologen, pp. 287–318. Hogrefe, Göttingen (2000)
Billiet, J., Meuleman, B.: Religious diversity in Europe and its relation to social attitudes and value orientations. Paper presented at the mini-conference on Index construction for cross cultural research, ESADE Business School, Universitat Ramon Llull, Barcelona, Spain, 13–15 February 2008
Bollen K.A.: Structural Equations with Latent Variables. Wiley, New York (1989)
Buchanan T.: Online assessment: desirable or dangerous?. Prof. Psychol. Res. Pract. 33, 148–154 (2002)
Buchanan T., Smith J.L.: Using the Internet for psychological research: personality testing on the World-Wide Web. Br. J. Psychol. 90, 125–144 (1999)
Byrne B.M.: Structural Equation Modeling with AMOS: Basic Concepts, Applications and Programming. Erlbaum, Mahwah (2001)
Byrne B.M., Shavelson R.J., Muthén B.: Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychol. Bull. 105, 456–466 (1989)
Chen F.F.: Sensitivity of goodness of fit indices to lack of measurement invariance. Struct. Eq. Model. 14, 464–504 (2007)
Davidov E., Meuleman B., Billiet J., Schmidt P.: Values and support for immigration. A cross-country comparison. Eur. Sociol. Rev. 24, 583–599 (2008a)
Davidov, E., Schmidt, P., Schwartz, S.H.: Bringing values back. In: The Adequacy of the European Social Survey to Measure Values in 20 Countries. Public Opin. Q. 72(3), 420–445 (2008b)
De Beuckelaer, A.: Measurement invariance issues in internationalmanagement research. Unpublished dissertation, Limburg, Netherlands (2005)
DeBeuckelaerA. Lievens F.: Measurement equivalence of paper-and-pencil and internet organisational surveys: a large scale examination in 16 countries. Appl. Psychol: Int. Rev. 58, 336–361 (2009)
Dillman D.A., Bowker D.K.: The web questionnaire challenge to survey methodologists. In: Reips, U.-D., Bosnjak, M. (eds) Dimensions of Internet Science, pp. 159–177. Papst, Lengerich (2001)
Fenlason K.J., Suckow-Zimberg K.: Online surveys. Critical issues in using the web to conduct surveys. In: Kraut, A.I. (eds) Getting Action from Organizational Surveys, pp. 183–212. Jossey-Bass, San Franciso (2006)
Ferrando P.J., Lorenzo-Seva U.: IRT-based factor analytic procedures for testing the equivalence of paper-and-pencil and internet-administered questionnaires. Psychol. Methods 10, 193–205 (2005)
Groves R.M.: Survey Errors a nd Survey Costs. Wiley, New York (1989)
Hauptmanns P.: Grenzen und Chancen von quantitativen Befragungen mit Hilfe des Internet. In: Batinic, B., Werner, A., Graf, L., Bandilla, W. (eds) Online Research: Methoden, Anwendungen und Ergebnisse, pp. 21–38. Hogrefe, Göttingen (1999)
Hitlin S., Piliavin J.A.: Values: reviving a dormant concept. Ann. Rev. Sociol. 30, 359–393 (2004)
Hu L.T., Bentler P.M.: Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct. Eq. Model. 6, 1–55 (1999)
Joinson A.: Social desirability, anonymity, and internet-based questionnaires. Behav. Res. Methods Instrum. Comp. 31(3), 433–438 (1999)
Jöreskog K.G.: Simultaneous factor analysis in several populations. Psychometrika 36, 409–426 (1971)
Joreskog K.G.: Testing structural equation models. In: Bollen, K.A., Long, J.S. (eds) Testing Structural Equation Models, Sage, Newburry Park (1993)
Kline R.B.: Principles and Practice of Structural Equation Modelling. Guilford Press, New York (1998)
Kluckhohn C.: Values and value-orientations in the theory of action: an exploration in definition and classification. In: Parsons, T., Shils, E. (eds) Towards a General Theory of Action, pp. 338–433. Harvard University Press, Cambridge, MA (1951)
Knoppen, D., Saris, W.: Measuring value orientations across nations: analysis of the quality of the european social survey for measuring values. Paper presented at the 2nd European Survey Research Association (ESRA) conference. Prague, Czech Republic, June 2007
Marsh H.W., Hau K.T., Wen Z.: In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing hu and bentler’s (1999) findings. Struct. Eq. Model. 11, 320–341 (2004)
Meredith W.: Measurement invariance, factor analysis and factorial invariance. Psychometrika 58, 525–543 (1993)
Ployhart R.E., Weekley J.A., Holtz B.C., Kemp C.: Web-based and paper-and-pencil testing of applicants in a proctored setting: are personality, biodata, and situational judgment tests comparable?. Pers. Psychol. 56, 733–752 (2003)
Preckel F., Thiemann H.: Online- versus paper-pencil version of a high potential intelligence test. Swiss J. Psychol. 62, 131–138 (2003)
Richman W.L., Kiesler S., Weisband S., Drasgow F.: A meta-analytic study of social desirability distortion in computer administered questionnaires, traditional questionnaires and interviews. J. Appl. Psychol. 84, 754–775 (1999)
Rietz R., Wahl S.: Comparison of psychologist’s self image and their image in the internet and in print. In: Batinic, B., Reips, U.D., Bosnjak, M. (eds) Online Social Sciences, pp. 135–150. Hogrefe & Huber Publishers, Gottingen (2002)
Rokeach M.: The Nature of Human Values. Wiley, New York (1973)
Sackmary, B.: Internet survey research: Practices, problems, and prospects. In: Proceedings of the American Marketing Association Summer Educator’s Conference, Boston. http://faculty.buffalostate.edu/sackmabd/sackmary/Ama98.pdf. Cited 2 February 2008 (1998)
Sagiv L., Schwartz S.H.: Value priorities and readiness for out-group social contact. J. Pers. Soc. Psychol. 69(3), 437–448 (1995)
Salzberger, T.: Statistically equivalent, yet different in meaning: different measurement model representations—shown by means of the AGGLO-Scale. In: Reed, P.W., Luxton, S.L., Shaw, M. R. (eds.) Proceedings of the ANZMEC 1997, Australia New Zealand Marketing Educators Conference, vol. 2, pp. 707–728, Department of Marketing, Monash University, Caulfield East, Vic., Australia (1997)
Saris W.E.: Measurement models in sociology and political science. In: Cudeck, R., Toit, S., Sörbom, D. (eds) Structural Equation Modeling: Present and Future, pp. 97–115. Scientific Software International, Chicago (2001)
Schnell R.: Nonresponse in Bevölkerungsumfragen. Ausmaß, Entwicklung und Ursachen. Leske + Budrich, Opladen (1997)
Schnell R., Hill P.B., Esser E.: Methoden der Empirischen Sozialforschung. Oldenburg, München (1999)
Schwartz S.H.: Universals in the content and structure of values: theory and empirical tests in 20 countries. In: Zanna, M. (eds) Advances in Experimental Social Psychology, pp. 1–65. Academic Press, New York (1992)
Schwartz S.: Beyond individualism/collectivism: new cultural dimensions of values. In: Kim, U., Triandis, H.C., Kagitcibasi, C., Choi, S.-C., Yoon, G. (eds) Individualism and Collectivism: Theory, Methods, and Applications, pp. 85–119. Sage, Newbury Park (1994)
Schwartz, S.H.: A proposal for measuring value orientations across nations. Chapter 7 in the ESS Questionnaire Development Report. http://www.europeansocialsurvey.org (2003)
Schwartz S.H.: Basic human values: their content and structure across countries. In: Tamayo, A., Porto, J.B. (eds) Valores e Comportamento nas Organizações, pp. 21–55. Vozes, Petrópolis (2005a)
Schwartz S.H.: Robustness and fruitfulness of a theory of universals in individual human values. In: Tamayo, A., Porto, J.B. (eds) Valores e Comportamento nas Organizações, pp. 21–55. Vozes, Petrópolis (2005b)
Schwartz S., Boehnke K.: Evaluating the structure of human values with confirmatory factor analysis. J. Res. Pers. 38, 230–255 (2004)
Schwartz S., Sagiv L.: Identifying culture-specifics in the content and structure of values. J. Cross-Cult. Psychol. 26(1), 92–116 (1995)
Sörbom D.: A general method for studying differences in factor means and factor structure between groups. Br. J. Math. Stat. Psychol. 27, 229–239 (1974)
Sörbom D.: An alternative to the methodology for analysis of covariance. Psychometrika 43, 381–396 (1978)
Steenkamp J.B., Baumgartner H.: Assessing measurement invariance in cross national consumer research. J. Consum. Res. 25, 78–90 (1998)
Vandenberg R.L., Lance C.E.: A review and synthesis of the measurement invariance literature: suggestions, practices, and recommendations for organizational research. Organ. Res. Methods 3(1), 4–70 (2000)
Weis C.H., Steinmetz P.: Marktforschung 5. Auflage. Friedrich Kiehl Verlag, Ludwigshafen (2002)
Wilhelm O., McKnight P.E.: Ability and achievement testing on the world wide web. In: Batinic, B., Reips, U.D., Bosnjak, M. (eds) Online Social Sciences, pp. 151–181. Hogrefe, Toronto (2002)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Davidov, E., Depner, F. Testing for measurement equivalence of human values across online and paper-and-pencil surveys. Qual Quant 45, 375–390 (2011). https://doi.org/10.1007/s11135-009-9297-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-009-9297-9