Abstract
Survey measures should be valid, reliable, and, on top of that, comparable when implemented in cross-cultural research. Therefore, survey measures should be pretested before the actual data collection. A valuable pretesting tool for cross-cultural research is web probing. Web probing implements probing techniques from cognitive interviewing in web survey and is particularly useful for the assessment of validity and comparability of survey measures. In this chapter, we discuss the origin of the web probing approach in cognitive interviewing, explain the approach and the different probing techniques (category-selection, specific, and comprehension probing), and illustrate the potential of web probing with different empirical examples. In addition, relevant methodological recommendations regarding the optimal implementation of web probing will be presented. We also comment on additional issues to consider when web probing is implemented in a cross-cultural context (e.g., translation, cultural expertise).
This is a preview of subscription content, log in via an institution to check access.
References
Aizpurua, E. (2020). Pretesting methods in cross-cultural research. In M. Sha & T. Gabel (Eds.), The essential role of language in survey research (pp. 129–150). RTI Press.
Anstötz, P., Schmidt, P., & Heyder, A. (2019). Wie valide ist die empirische Messung der Through- und Outputlegitimität politischer Systeme? In C. Wiesner & P. Harfst (Eds.), Legitimität und Legitimation (pp. 33–55). Springer Fachmedien. https://doi.org/10.1007/978-3-658-26558-8_3
Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311. https://doi.org/10.1093/poq/nfm006
Behr, D. (2015). Translating answers to open-ended survey questions in cross-cultural research: A case study on the interplay between translation, coding, and analysis. Field Methods, 27(3), 284–299. https://doi.org/10.1177/1525822X14553175
Behr, D. (2017). Assessing the use of back translation: The shortcomings of back translation as a quality testing method. International Journal of Social Research Methodology, 20(6), 573–584. https://doi.org/10.1080/13645579.2016.1252188
Behr, D. (2018). Translating questionnaires for cross-national surveys: A description of a genre and its particularities based on the ISO 17100 categorization of translator competences. Translation & Interpreting, 10(2), 5–20. https://doi.org/10.12807/ti.110202.2018.a02
Behr, D. (2023). Translating questionnaires. In C. U. Krägeloh, M. Alyami, & O. N. Medvedev (Eds.), International handbook of behavioral health assessment. Springer. https://doi.org/10.1007/978-3-030-89738-3_2-1
Behr, D., & Braun, M. (2023). How does back translation fare against team translation? An experimental case study in the language combination English-German. Journal of Survey Statistics and Methodology, 11(2), 285–315. https://doi.org/10.1093/jssam/smac005
Behr, D., Kaczmirek, L., Bandilla, W., & Braun, M. (2012). Asking probing questions in web surveys: Which factors have an impact on the quality of responses? Social Science Computer Review, 30(4), 487–498. https://doi.org/10.1177/0894439311435305
Behr, D., Braun, M., Kaczmirek, L., & Bandilla, W. (2013). Testing the validity of gender ideology items by implementing probing questions in web surveys. Field Methods, 25(2), 124–141. https://doi.org/10.1177/1525822X12462525
Behr, D., Braun, M., Kaczmirek, L., & Bandilla, W. (2014). Item comparability in cross-national surveys: Results from asking probing questions in cross-national web surveys about attitudes towards civil disobedience. Quality & Quantity, 48(1), 127–148. https://doi.org/10.1007/s11135-012-9754-8
Behr, D., Braun, M., & Dorer, B. (2016). Measurement instruments in cross-national surveys (version 2.0) (GESIS survey guidelines). GESIS – Leibniz-Institut für Sozialwissenschaften. https://doi.org/10.15465/gesissg_en_006
Behr, D., Meitinger, K., Braun, M., & Kaczmirek, L. (2017). Web probing: Implementing probing techniques from cognitive interviewing in web surveys with the goal to assess the validity of survey questions (GESIS – Survey guidelines). Retrieved from https://www.gesis.org/gesis-survey-guidelines/instruments/qualitaet-von-umfragedaten/web-probing
Behr, D., Meitinger, K., Braun, M., & Kaczmirek, L. (2020). Cross-national web probing: An overview of its methodology and its use in cross-national studies. In P. C. Beatty, D. Collins, L. Kaye, J.-L. Padilla, G. B. Willis, & A. Wilmot (Eds.), Advances in questionnaire design, development, evaluation and testing (pp. 521–543). Wiley. https://doi.org/10.1002/9781119263685.ch21
Braun, M., Behr, D., & Kaczmirek, L. (2013). Assessing cross-national equivalence of measures of xenophobia: Evidence from probing in web surveys. International Journal of Public Opinion Research, 25(3), 383–395. https://doi.org/10.1093/ijpor/eds034
Braun, M., Behr, D., Kaczmirek, L., & Bandilla, W. (2014). Evaluating cross-national item equivalence with probing questions in web surveys. In U. Engel, B. Jann, P. Lynn, A. Scherpenzeel, & P. Sturgis (Eds.), Improving survey methods: Lessons from recent research (pp. 184–200). Routledge.
Braun, M., Behr, D., & Díez Medrano, J. (2018). What do respondents mean when they report to be “citizens of the world”? Using probing questions to elucidate international differences in cosmopolitanism. Quality & Quantity, 52(3), 1121–1135. https://doi.org/10.1007/s11135-017-0507-6
Braun, M., Behr, D., Meitinger, K., Raiber, K., & Repke, L. (2019). Using web probing to elucidate respondents’ understanding of ‘minorities’ in cross-cultural comparative research. ASK: Research and Methods, 28(1), 3–20. https://doi.org/10.18061/ask.v28i1.0001
Braun, M., Meitinger, K. M., & Behr, D. (2020). Combining quantitative experimental data with web probing: The case of individual solutions for the division of labor between both genders. Methods, Data, Analyses, 14(2), 215–240. https://doi.org/10.12758/mda.2020.03
Caspar, R., Peytcheva, E., Yan, T., Lee, S., Liu, M., & Hu, M. (2016). Pretesting (cross-cultural survey guidelines). Retrieved from https://ccsg.isr.umich.edu/chapters/pretesting
Conrad, F. G., Schober, M. F., Jans, M., Orlowski, R. A., Nielsen, D., & Levenstein, R. (2015). Comprehension and engagement in survey interviews with virtual agents. Frontiers in Psychology, 6, 1578. https://www.frontiersin.org/articles/10.3389/fpsyg.2015.01578
DeMaio, T. J., & Landreth, A. (2004). Do different cognitive interview techniques produce different results? In S. Presser, J. M. Rothgeb, M. P. Couper, J. T. Lessler, E. Martin, J. Martin, & E. Singer (Eds.), Methods for testing and evaluating survey questionnaires (pp. 89–108). Wiley. https://doi.org/10.1002/0471654728.ch5
Difallah, D., Filatova, E., & Ipeirotis, P. (2018) Demographics and dynamics of mechanical Turk worker. In Proceedings of the eleventh ACM international conference on web search and data mining (pp. 135–143). https://doi.org/10.1145/3159652.3159661
Dorer, B. (2021). Translating answers to open-ended questions in a multilingual survey context: Challenges and recommendations. trans-kom, 14(2), 226–256. http://www.trans-kom.eu/bd14nr02/trans-kom_14_02_03_Dorer_Questions.20211202.pdf
Edgar, J., Murphy, J., & Keating, M. (2016). Comparing traditional and crowdsourcing methods for pretesting survey questions. SAGE Open, 6(4). https://doi.org/10.1177/2158244016671770
Fitzgerald, R., Widdop, S., Gray, M., & Collins, D. (2011). Identifying sources of error in cross-national questionnaires: Application of an error source typology to cognitive interview data. Journal of Official Statistics, 27(4), 569–599. https://openaccess.city.ac.uk/id/eprint/1160/1/Identifying%20sources%20of%20error%20-%20JOS%20article.pdf
Fowler, S., & Willis, G. B. (2020). The practice of cognitive interviewing through web probing. In P. Beatty, D. Collins, L. Kaye, J. L. Padilla, G. Willis, & A. Wilmot (Eds.), Advances in questionnaire design, development, evaluation and testing (pp. 451–469). John Wiley & Sons. https://doi.org/10.1002/9781119263685.ch18
Gavras, K., Höhne, J. K., Blom, A., & Schoen, H. (2022). Innovating the collection of open-ended answers: The linguistic and content characteristics of written and oral answers to political attitude questions. Journal of the Royal Statistical Society, Series A: Statistics in Society, 185(3), 872–890. https://doi.org/10.1111/rssa.12807
Geisen, E., & Murphy, J. (2020). A compendium of web and mobile survey pretesting methods. In P. Beatty, D. Collins, L. Kaye, J. L. Padilla, G. Willis, & A. Wilmot (Eds.), Advances in questionnaire design, development, evaluation and testing (pp. 287–314). John Wiley & Sons. https://doi.org/10.1002/9781119263685.ch12
Hadler, P. (2021). Question order effects in cross-cultural web probing: Pretesting behavior and attitude questions. Social Science Computer Review, 39(6), 1292–1312. https://doi.org/10.1177/0894439321992779
Hadler, P., Neuert, C. E., Ortmanns, V., & Stiegler, A. (2022). Are you…? Asking questions on sex with a third category in Germany. Field Methods, 34(2), 91–107. https://doi.org/10.1177/1525822X211072326
Harkness, J. A. (2003). Questionnaire translation. In J. A. Harkness, F. J. R. van de Vijver, & P. P. Mohler (Eds.), Cross-cultural survey methods (pp. 35–56). Wiley.
Harkness, J. A. (2008). Comparative survey research: Goal and challenges. In E. D. de Leeuw, J. J. Hox, & D. A. Dillman (Eds.), International handbook of survey methodology (pp. 56–77). Lawrence Erlbaum Associates.
Harkness, J. A., Villar, A., & Edwards, B. (2010). Translation, adaptation, and design. In J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson, L. Lyberg, P. Ph. Mohler, B-E. Pennell, & T. W. Smith (Eds.), Survey methods in multinational, multiregional, and multicultural contexts (pp. 117–140). Wiley. https://doi.org/10.1002/9780470609927.ch7.
He, J., & van de Vijver, F. (2012). Bias and equivalence in cross-cultural research. Online Readings in Psychology & Culture, 2(2), 1–19. https://doi.org/10.9707/2307-0919.1111
Hyukjun, G., & Schonlau, M. (2023). Automated classification for open-ended questions with BERT. Journal of Survey Statistics and Methodology, smad015. https://doi.org/10.1093/jssam/smad015
Kaczmirek, L., Meitinger, K., & Behr, D. (2017). Higher data quality in web probing with EvalAnswer: A tool for identifying and reducing nonresponse in openended questions (GESIS Papers, 2017/01). GESIS – Leibniz Institut für Sozialwissenschaften. https://doi.org/10.21241/ssoar.51100
Kunz, T., & Meitinger, K. (2022). A comparison of three designs for list-style open-ended questions in web surveys. Field Methods, 34(4), 303–317. https://doi.org/10.1177/1525822X221115831
Lee, S., McClain, C., Behr, D., & Meitinger, K. (2020). Exploring mental models behind self-rated health and subjective life expectancy through web probing. Field Methods, 32(3), 309–326. https://doi.org/10.1177/1525822X20908575
Leitgöb, H., Seddig, D., Asparouhov, T., Behr, D., Davidov, E., De Roover, K., Jak, S., Meitinger, K., Menold, N., Muthén, B., Rudnev, M., Schmidt, P., & van de Schoot, R. (2023). Measurement invariance in the social sciences: Historical development, methodological challenges, state of the art, and future perspectives. Social Science Research, 110, 102805. https://doi.org/10.1016/j.ssresearch.2022.102805
Lenzner, T., & Neuert, C. E. (2017). Pretesting survey questions via web probing – Does it produce similar results to face-to-face cognitive interviewing? Survey Practice, 10(4) https://doi.org/10.29115/SP-2017-0020
Meitinger, K. (2017). Necessary but insufficient: Why measurement invariance tests need online probing as a complementary tool. Public Opinion Quarterly, 81(2), 447–472. https://doi.org/10.1093/poq/nfx009
Meitinger, K. (2018). What does the general national pride item measure? Insights from web probing. International Journal of Comparative Sociology, 59(5–6), 428–450. https://doi.org/10.1177/0020715218805793
Meitinger, K., & Behr, D. (2016). Comparing cognitive interviewing and online probing: Do they find similar results? Field Methods, 28(4), 363–380. https://doi.org/10.1177/1525822X15625866
Meitinger, K., & Kunz, T. (2022). Visual design and cognition in list-style open-ended questions in web probing. Sociological Methods & Research. https://doi.org/10.1177/00491241221077241
Meitinger, K., Braun, M., & Behr, D. (2018). Sequence matters in web probing: The impact of the order of probes on response quality, motivation of respondents, and answer content. Survey Research Methods, 12(2), 103–120. https://doi.org/10.18148/srm/2018.v12i2.7219
Meitinger, K., Behr, D., & Braun, M. (2021). Using apples and oranges to judge quality? Selection of appropriate cross-national indicators of response quality in open-ended questions. Social Science Computer Review, 39(3), 434–455. https://doi.org/10.1177/0894439319859848
Meitinger, K., Toroslu, A., Raiber, K., & Braun, M. (2022). Perceived burden, focus of attention, and the urge to justify: The impact of the number of screens and probe order on the response behavior of probing questions. Journal of Survey Statistics and Methodology, 10(4), 923–944. https://doi.org/10.1093/jssam/smaa043
Miller, K. (2019). Conducting cognitive interviewing studies to examine survey question comparability. In T. P. Johnson, B.-E. Pennell, I. A. L. Stoop, & B. Dorer (Eds.), Advances in comparative survey methods: Multinational, multiregional, and multicultural contexts (3MC) (1st ed., pp. 203–225). Wiley. https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118884997.ch10
Miller, K., Fitzgerald, R., Padilla, J.-L., Willson, S., Widdop, S., Caspar, R., Dimov, M., Gray, M., Nunes, C., Prüfer, P., Schöbi, N., & Schoua-Glusberg, A. (2011). Design and analysis of cognitive interviews for comparative multinational testing. Field Methods, 23(4), 379–396. https://doi.org/10.1177/1525822X11414802
Neuert, C., & Lenzner, T. (2021). Effects of the number of open-ended probing questions on response quality in cognitive online pretests. Social Science Computer Review, 39(3), 456–468. https://doi.org/10.1177/0894439319866397
Neuert, C., & Lenzner, T. (2023). Design of multiple open-ended probes in cognitive online pretests using web probing. Survey Methods: Insights from the Field (SMIF). https://doi.org/10.13094/SMIF-2023-00005
Neuert, C., Meitinger, K., Behr, D., & Schonlau, M. (2021). Editorial: The use of open-ended questions in surveys. Methods, Data, Analyses, 15(1), 3–6. https://mda.gesis.org/index.php/mda/article/view/366/280
Padilla, J. L., & Leighton, J. P. (2017). Cognitive interviewing and think aloud methods. In B. D. Zumbo & A. M. Hubley (Eds.), Understanding and investigating response processes in validation research (pp. 211–228). Springer. https://doi.org/10.1007/978-3-319-56129-5_12
Pennell, B.-E., Hibben, K. C., Lyberg, L. E., Mohler, P. P., & Worku, G. (2017). A total survey error perspective on surveys in multinational, multiregional, and multicultural contexts. In P. P. Biemer, E. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L. E. Lyberg, C. Tucker, & B. T. West (Eds.), Total survey error in practice (pp. 179–201). John Wiley & Sons. https://onlinelibrary.wiley.com/doi/abs/10.1002/9781119041702.ch9
Prüfer, P., & Rexroth, M. (2005). Kognitive interviews. ZUMA How-to-Reihe, 15. https://www.ssoar.info/ssoar/handle/document/20147
Revilla, M., Couper, M. P., Bosch, O. J., & Asensio, M. (2020). Testing the use of voice input in a smartphone web survey. Social Science Computer Review, 38(2), 207–224. https://doi.org/10.1177/0894439318810715
Scanlon, P. (2019). The effects of embedding closed-ended cognitive probes in a web survey on survey response. Field Methods, 31(4), 328–343. https://doi.org/10.1177/1525822X19871546
Scanlon, P. (2020). Using targeted embedded probes to quantify cognitive interviewing findings. In P. Beatty, D. Collins, L. Kaye, J. L. Padilla, G. Willis, & A. Wilmot (Eds.), Advances in questionnaire design, development, evaluation and testing (pp. 427–449). Wiley. https://doi.org/10.1002/9781119263685.ch17
Scholz, E., Dorer, B., & Züll, C. (2022). Coding issues of open-ended questions in a cross-cultural context. International Journal of Sociology, 52(1), 78–96. https://doi.org/10.1080/00207659.2021.2015664
Schulz, S., Meitinger, K., Braun, M., & Behr, D. (2018). Who’s bad? Eine Analyse zur internationalen Vergleichbarkeit von Maßen krimineller Einstellungen mittels des Web-Probing Ansatzes. In K. Boers & M. Schaerff (Eds.), Kriminologische Welt in Bewegung (Neue Kriminologische Schriftenreihe 117) (pp. 406–417). Forum Verlag. https://d-nb.info/1171373074
Silber, H., Zuell, C., & Kuehnel, S. (2020). What can we learn from open questions in surveys? A case study on non-voting reported in the 2013 German longitudinal election study. Methodology, 16(1), 41–58. https://doi.org/10.5964/meth.2801
Singer, E., & Couper, M. P. (2017). Some methodological uses of responses to open questions and other verbatim comments in quantitative surveys. Methods, Data, Analyses, 11(2), 115–134. https://doi.org/10.12758/mda.2017.01
Smith, T. W. (2020). Optimizing questionnaire design in cross-national and cross-cultural surveys. In P. Beatty, D. Collins, L. Kaye, J. L. Padilla, G. Willis, & A. Wilmot (Eds.), Advances in questionnaire design, development, evaluation and testing (pp. 471–492). Wiley. https://doi.org/10.1002/9781119263685.ch19
Toepoel, V., Mathon, K., Tussenbroek, P., & Lugtig, P. (2021). Probing in online mixed-device surveys: Is a research messenger layout more effective than a traditional online layout, especially on mobile devices? Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 151(1), 74–95. https://doi.org/10.1177/07591063211019953
Tourangeau, R., Rips, L., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322
van de Vijver, F. J. R., & Leung, K. (2021). Methodological concepts in cross-cultural research. In V. H. Fetvadjiev, J. He, & J. R. J. Fontaine (Eds.), Methods and data analysis for cross-cultural research (2nd ed., pp. 4–9). Cambridge University Press. https://doi.org/10.1017/9781107415188.003
van de Vijver, F. J. R., & Poortinga, Y. H. (1997). Towards an integrated analysis of bias in cross-cultural assessment. European Journal of Psychological Assessment, 13(1), 29–37. https://doi.org/10.1027/1015-5759.13.1.29
Willis, G. B. (2005). Cognitive interviewing. A tool for improving questionnaire design. Sage.
Willis, G. B. (2015). The practice of cross-cultural cognitive interviewing. Public Opinion Quarterly, 79(S1), 359–395. https://doi.org/10.1093/poq/nfu092
Willis, G. B., & Miller, K. (2011). Cross-cultural cognitive interviewing: Seeking comparability and enhancing understanding. Field Methods, 23(4), 331–341. https://doi.org/10.1177/1525822X11416092
Züll, C. (2016). Open-ended questions (Version 2.0) (GESIS survey guidelines). GESIS – Leibniz-Institut für Sozialwissenschaften. https://doi.org/10.15465/gesis-sg_en_002
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 Springer Nature Switzerland AG
About this entry
Cite this entry
Meitinger, K., Neuert, C., Behr, D. (2024). Cross-Cultural Web Probing. In: Krägeloh, C.U., Alyami, M., Medvedev, O.N. (eds) International Handbook of Behavioral Health Assessment. Springer, Cham. https://doi.org/10.1007/978-3-030-89738-3_3-1
Download citation
DOI: https://doi.org/10.1007/978-3-030-89738-3_3-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89738-3
Online ISBN: 978-3-030-89738-3
eBook Packages: Springer Reference Behavioral Science and PsychologyReference Module Humanities and Social SciencesReference Module Business, Economics and Social Sciences