Introduction

Play of chance is one major drawback of observational epidemiology. To address this, there has been an increasing trend towards the establishment of large epidemiological studies involving hundreds of thousands of participants. Such observational studies require very heavy logistic resources related to printing and mailing of questionnaires, as well as data entry and cleaning of collected information. Measurement error in the assessment of lifestyle also decreases the chances of observing the true associations. Epidemiology then faces the challenge to increase power thanks to big sample size and high quality data in the most cost-effective way. Web-based studies nowadays raise a major interest worldwide (Ekman and Litton 2007; Schatzkin et al. 2009) as they hold improvement of all steps of the long process involved in observational studies: from reduction of missing or aberrant data due to integrated controls to multimedia support and cost reduction.

Internet access constantly increases around the world, in most socio-demographic groups. France had 35 millions of Internet users in February 2010 (L’audience de l’Internet en France Médiamétrie, http://www.mediametrie.fr/internet/) and came at the top of the European classification of Internet usage according to the EIAA in 2005 (European Interactive Advertising Association Internet, http://www.eiaa.net/news/eiaa-articles-details.asp?lang=1&id=79). About 66% of the population aged more than 11 years were connected to Internet during the previous month (L’audience de l’Internet en France Médiamétrie). Besides, 27% of the Internet users belong to low social classes (Profil sociodemographique des internautes Ipsos Media, http://www.ipsos.fr/canalipsos/poll/8351.asp, 2007) and one Internet user in four is more than 55 years old (European Interactive Advertising Association). Senior Internet users in France have the highest score in Internet connection in Europe (European Interactive Advertising Association) making France very attractive to launch an ambitious Internet-based study.

In spring 2009, the NutriNet-Santé study was launched, the first web-based prospective observational cohort study worldwide, with the objective of recruiting 500,000 volunteers aged ≥18 years (recruitment planned for 5 years). It aims to investigate the relationships between lifestyle and health (incidence of ischaemic heart disease, cancers, overall mortality, etc.) (Etude NutriNet-Santé Website https://www.etude-nutrinet-sante.fr, 2009). Five main questionnaires are administrated via Internet at baseline to assess socio-demographic and economic characteristics, anthropometry, health status, physical activity and diet.

Socioeconomic status is one important risk factor for chronic diseases (Clark et al. 2009; Ward et al. 2004). In addition, it has been associated with food habits (Drewnowski 2009), obesity (McLaren 2007), physical activity (Bertrais et al. 2004) and is therefore a major confounder in all observational studies evaluating the relation between lifestyle and health. Assessing the right information as well as keeping it up to date during follow-up is of major importance. However, few controlled studies have investigated the potential differential effect of questionnaire administration modes on socio-demographic and economic variables reported in adults (Graham and Papandonatos 2008; Greene et al. 2008; Rankin et al. 2008). None of them evaluated the robustness of an entire set of socio-demographic and economic variables. Integrated controls programmed in the web-based version provide more guidance compared to the paper version, and can help subjects to remember and answer the questions correctly. Reduction of filling duration due to automatic branching could also decrease the misreporting related to lassitude in the web-based version. On the other hand, participants are more used to fill paper than web-based questionnaires. Therefore, agreement between web-based and paper questionnaires needed to be assessed before using this promising tool.

The objective of the present study was to evaluate the agreement between the web-based version of the socio-demographic and economic questionnaire of NutriNet-Santé with the traditional paper version. Both versions were self-administered in a sample of 170 French subjects.

Methods

Study population

Volunteers were recruited in January 2009 via a call for participation among subjects of the SU.VI.MAX. cohort both verbally (during the annual meeting of subjects) and by e-mail (collected by paper questionnaires during the SU.VI.MAX. follow-up for 5,949 subjects out of 13,017). The design, methods and rationale of this cohort have been described elsewhere (Hercberg et al. 1998). A total of 1,090 persons volunteered and were available for participating in the NutriNet-Santé validation studies. Among them, 170 were randomly selected to test the socio-demographic and economic questionnaire. All procedures were approved by the Ethical Committee for Studies with Human Subjects and the Commission Nationale Informatique et Liberté (CNIL No. 334641 for the SU.VI.MAX. study and No. 908450 for the validation studies). Electronic informed consent was obtained from all subjects.

Socio-demographic and economic questionnaire (web-based and traditional paper versions)

The self-administered web-based socio-demographic and economic questionnaire of NutriNet-Santé includes 30 questions divided into sub-questions (resulting in 70 variables), dealing with basic socio-demographic variables, current job status, occupational categories, income, educational attainment, household composition and tobacco consumption [http://media.etude-nutrinet-sante.fr/questionnaires/questionnaires_nutrinet_papier.zip, NutriNet-Santé questionnaires (in French), 2009]. As some questions depend on the answers given previously, subjects are automatically directed to the appropriate question according to their previous answers. The web-based version did not allow missing values (all items needed to be completed to access the next screen) and boundary controls were included to avoid implausible answers. When these conditions were not respected, the subject was alerted by a message that indicated the nature of the problem and was requested to correct it before going back to the completion of the questionnaire. A text help section was available and subjects were allowed to fill the questionnaire in one or several go. Subjects also had access to video explanations about the filling of the questionnaire through the Internet and a supporting video was displayed when the subject attained half of the questionnaire to encourage him/her to fill in the entire questionnaire. In addition, a recapitulation form helping them to see if they committed any mistake was displayed once the questionnaire was filled.

For the present study, a traditional paper version of this questionnaire was developed. As for the web-based version, participants could fill it in one or several attempts and had access to the text help section. Instead of an automatic direction to the appropriate question, the conditional branching in the paper version was indicated by written instructions (e.g. “if yes, then go to question no. 2.3, and if no, go to no. 2.2”). The video explanations and the supporting video as well as the recap form could not be implemented in the paper version.

Study design

Subjects were randomly assigned into two groups (n = 85 each) according to the order of administration: group 1, paper version first (the login and password were provided to the subjects for access to the web-based version when we received their completed paper questionnaire by regular mail); group 2, web-based version first (the paper version was mailed to the subjects when they ended the completion of the web-based version). Answers of the paper version were captured by two independent keyboard operators. When an inconsistency between both operators was detected, data were checked on the paper questionnaire and the right answer was retained. All 170 subjects had to fill in both versions of the questionnaire.

Satisfaction questionnaire

After completing both versions of the socio-demographic and economic questionnaire, the subjects were invited to fill in a web-based satisfaction questionnaire about their self-estimated situation towards the Internet (self-evaluated computer and web knowledge, web access, type of connection), durations of filling in the respective versions of the questionnaire, opinions about each version (aesthetic, fluency of use, etc.) and preferred version.

Statistical analyses

Out of 70 variables, 32 were too specific, resulting in too small a sample size (n < 30) (i.e. 7 variables related to adoptive children; 8 variables related to third and fourth children; 8 variables related to the number of cigarette, cigar and pipe smoked in previous, current or occasional smokers; 3 variables related to the number of children living in the house; 6 free format variables corresponding to “other–please specify” solutions) and 2 had substantial imbalances in the table’s marginal totals (all subjects answered yes to the two questions “Did you ever work?” and “Did your partner ever worked?”), making kappa estimates unreliable. Therefore, 36 variables were analysed in the present study.

First, agreement between both versions of the socio-demographic and economic questionnaire was assessed by intraclass correlations (ICC) for continuous variables (Shrout and Fleiss 1979), kappas (Feinstein and Cicchetti 1990) for categorical variables and their corresponding 95% confident intervals (CI).

ICC measures the correlation between a single rating on a continuous measure using the Internet-based questionnaire, with a single rating of the same measure obtained with the paper questionnaire. Reliability above 80% is usually considered as satisfactory in method comparisons, with 70% considered an acceptable value. Since ICC is based on mean-centred versions of the variables of interest, it is insensitive to participants’ tendency to provide consistently higher responses on one survey instrument than another. Therefore, we further implemented Wilcoxon sign rank tests to detect the presence of a systematic location difference between web-based and paper questionnaires.

Kappa measures the level of between-method agreement beyond that which can be ascribed to chance. Standard cutoffs for measuring agreement have been established by Landis and Koch (1977): 0.80–1.00 = almost perfect, 0.60–0.80 = substantial, 0.40–0.60 = moderate, 0.20–0.40 = fair, 0.00–0.20 = slight and <0.00 = poor. Weighted kappas were used for ordinal variables as well as continuous variables (one category per integer). ICC, kappas and Wilcoxon sign rank tests were calculated overall and stratified by administration order group (paper or web first), age, gender and self-estimated web knowledge (experienced or novice).

Second, we quantified the number of data entry mistakes and missing (data that should have been filled but did not), inconsistent (data that should not have been filled according to the branching logic) or aberrant data (out of the pre-defined plausible range) inherent in the paper version (nonexistent in the web-based version by construction, due to integrated controls). Third, all parameters of the satisfaction questionnaire (durations of filling and appreciations) were compared between the web-based and the paper versions of the socio-demographic and economic questionnaire by Mac–Nemar chi-square tests (for 2 × 2 tables) or Bowker’s test of symmetry (generalisation of the Mac–Nemar chi-square tests for more than 2 categories).

Finally, we evaluated the additional cost related to the paper version of the socio-demographic and economic questionnaire (printing, mailing and staff for data entry, nonexistent in the web-based version) and the cost related to the development and support of the web tool for 170 and 500,000 subjects. For all analyses, the significance level was two sided and set at 0.05. All statistical analyses were performed using SAS software (version 9.1, SAS Institute Inc., Cary, NC, USA).

Results

Characteristics of the study population

From the 170 initially selected subjects, 23 were excluded: 15 subjects gave no returns despite several reminders, 5 had misunderstood the modalities of the study and withdrew their participation, 2 faced connection issues and 1 stopped because he judged the web system to be too complex for him. These subjects were not different from the subjects included in terms of age, percentage of male, educational levels and occupational categories (data not shown). Thus, 147 subjects remained for the analyses (76 completed the paper version first and 71 the web-based version first); among them, 128 subjects filled in the satisfaction questionnaire. The median number of days between the filling of both versions was 9 (minimum = 2, maximum = 61 days). Socio-demographic and web characteristics of the study population were well-balanced between the administration order groups (Table 1).

Table 1 Characteristics of the subjects according to administration order in the NutriNet-Santé validation study, France, 2009
Full size table

Agreement between the web-based and the paper versions

All ICC (Table 2) and kappa coefficients (Table 3) were superior to 0.80 except for the type of employment contract of the participant and his/her partner for which kappa of 0.76 [0.63–0.90] and 0.79 [0.77–0.92] were observed, respectively. Using weighted kappa coefficients instead of ICC did not substantially change these results. The agreement was superior to 0.80 in 13 variables and superior to 0.60 in the two remaining variables (duration of unemployment and cumulative duration of passive smoking at home, data not tabulated). No systematic location difference between the two versions was observed for continuous variables (according to the Wilcoxon sign rank tests, data not tabulated) except for the year of tobacco cessation and the tobacco consumption duration for which significant higher values were observed in the paper version compared to the web-based version (mean differences equal 1.4 and 0.9 years, respectively; p = 0.02 for both variables). After stratification (according to administration order group, age, gender and web knowledge), these differences remained only in women and subjects older than 60 years.

Table 2 Intraclass correlation between the web-based and the paper versions in the NutriNet-Santé validation study, France, 2009
Full size table
Table 3 Kappa statistic between the web-based and the paper versions in the NutriNet-Santé validation study, France, 2009
Full size table

Despite a high consistency between the subject’s ranking (assessed by both ICC and weighted kappas) in the total sample, a substantial number of discordant values between paper-based and web-based questionnaires were observed for the age of the first employment, the duration of unemployment, the number of cigarettes smoked per day among former smokers, the year of tobacco start and quit and the cumulative durations of tobacco use, passive smoking at home and passive smoking at work (Table 2).

When looking at stratified analyses, kappas and ICC were similar across all stratification groups with few exceptions: a lower concordance between the web-based and the paper versions was observed for: (1) the age at first employment in subjects ≥60 years compared to subjects younger than 60 years and in subjects with the paper version first compared to those with the web-based version first; (2) the unemployment duration in women compared to men and in subjects with the paper version first compared to those with the web-based version first; (3) the number of cigarettes smoked per day in former smokers younger than 60 years compared to older former smokers; (4) the year of tobacco cessation in women compared to men; and (5) the duration of passive smoking at home in men compared to women and in web experienced compared to web novice subjects (data not tabulated). However, these lower ICC were still superior to 0.70 except for the duration of unemployment in women (ICC = 0.59).

Data entry mistakes and missing, inconsistent or aberrant data in the paper version

In the paper version, there were 553 missing values (2% of total entries, 145 subjects with at least one missing value), 24 inconsistent data (0.09% of total entries, 24 subjects with at least one) and 8 aberrant data (0.03% of total entries, 7 subjects with at least one), not existing in the web-based version due to integrated control. The two keyboard operators combined made 472 data entry errors (0.85% of entries, 120 subjects with at least one). The maximum number of aberrant data was observed for starting smoking age (5 subjects out of the pre-defined plausible range) (data not tabulated).

Duration, satisfaction and cost

Within the 128 subjects who completed the satisfaction questionnaire, duration and easiness of filling were more favourably judged for the web-based version, compared to the paper version (p = 0.009 and 0.046, respectively) (Table 4). Subjects consulted more often the help section for the paper version than for the web-based version (p = 0.0006). The web-based version was preferred to the paper version by 93.7% of the subjects (data not tabulated).

Table 4 Comparison of duration and satisfaction between the web-based and the paper versions of the NutriNet-Santé socio-demographic and economic questionnaire in the NutriNet-Santé validation study, France, 2009
Full size table

Subjects were also asked about the features available in the web-based version (data not tabulated). Most of them (95.3%) filled out the web-based questionnaire in one go and 77.3% consulted the recapitulation form. Among them, 18.0% declared that this helped them to pick up a mistake and correct it. Besides, 95.3% of the subjects found the web-based interface user-friendly and 89.0% appreciated the mid-questionnaire support message. Finally, 14.8% of subjects viewed the help video; all of them were entirely or rather satisfied.

The additional cost related to the printing (0.33 €; US $0.48/questionnaire), mailing for sending (1.67 €; US $2.43/questionnaire) and returning the questionnaire (1.71 €; US $2.49/questionnaire) and double data entry (12.76 €; US $18.56/questionnaire calculated on the basis of the cost for one full-time equivalent: 1,913 €/month, including charges) of the paper version reached 2,800 € (US $4,072) and 8,235,000 € (US $11,975,502) for 170 and 500,000 subjects (i.e. the targeted effective for the NutriNet-Santé study), respectively. Our web-based tool was developed by professional web-designers in 9 months. It cost 150,000 € (US $213,000) for the whole NutriNet-Santé study process (not only for the socio-demographic and economic questionnaire) including secure inscription system, development and administration of baseline questionnaires, monitoring of the cohort (e-mails to the subjects, newsletter, etc.), equipment and hosting. Cost of an additional questionnaire equivalent to the socio-demographic and economic one would cost about 5,000 € (US $7,100; 4 days of development).

Discussion

The present study showed a strong agreement between the paper and the web-based versions of a socio-demographic and economic questionnaire. Furthermore, agreement was similar according to age, sex, web knowledge and order of administration for most variables. These results are consistent with those we previously observed for the anthropometric questionnaire (Touvier et al. 2010). Furthermore, many errors (data entry mistakes and missing, inconsistent or aberrant values) inherent in the paper version were avoided in the web-based version, due to automatic controls. The web-based version was highly appreciated and permitted considerable savings on logistic costs.

Socio-demographic and economic characteristics such as age, educational level, marital or employment status are very reliable variables and were, as expected, strongly concordant between the two methods of administration. No previous study evaluated the robustness of an entire set of socio-demographic and economic variables between web-based and traditional paper versions of a self-administered questionnaire. However, in agreement with our results, the reliability of self-reported income, considered as a sensitive question, was almost perfect in 319 American subjects who completed both a telephone interview and a web-based questionnaire 2 days apart (Graham and Papandonatos 2008). This result was not modified according to ethnic minorities or income levels. Furthermore, the agreement between residential history reported in two web-based questionnaires was higher than the one reported in two telephone interviews completed several weeks apart (Rankin et al. 2008).

More literature exists for the consumption of tobacco (McCabe et al. 2002; Pealer et al. 2001; Wang et al. 2005). A lot of them were performed in adolescents or young adults, a key population to evaluate whether the administration mode could influence the tendency to report tobacco consumption as well as more sensitive health compromising behaviours such as drug consumption. The concordance between responses of subjects randomly allocated in a web-based survey with those of subjects allocated in a paper survey was satisfying in general (McCabe et al. 2002; Pealer et al. 2001; Wang et al. 2005). Some studies even suggested that missing answers for socially threatening items were less frequent (Pealer et al. 2001) and that subjects were more likely to report using substances such as alcohol, tobacco and amphetamines (Wang et al. 2005) in the web-based survey than in the paper survey.

In our study, all tobacco consumption variables showed close agreement between both versions in the global sample (according to ICC and kappas). This was in agreement with previous results observed in current adults smokers showing that reliability between tobacco consumption history and number of cigarettes smoked per day reported in an initial survey with those reported in a survey performed several weeks apart was similar using either a web-based questionnaire or a telephone interview (Rankin et al. 2008). Furthermore, looking at the year of tobacco start, which is one of the most salient events, the agreement between the web-based and paper-based questionnaires administered a few days apart in our study (ICC = 0.97) was superior to the agreement between two identical web-based questionnaires administered 2 years apart (ICC = 0.81) (Brigham et al. 2009).

However, although still high, the concordance between both versions was a bit lower in few sub-groups for some variables involving an important memory recall. The concordance between the two versions was lower for the number of cigarettes smoked per day in former smokers younger than 60 years, for the year of tobacco cessation in women and for the duration of passive smoking at home in men. Furthermore, women and older subjects reported higher tobacco duration in the paper version than in the web-based version. This is in agreement with previous result from Stanton et al. (2007) who found that tobacco status misreport was more frequent in older subjects and when the earlier smoking was less frequent/intense. These types of information are cognitively complex to remember and to estimate. The smaller agreement observed for these variables could be associated with memory bias (Hammersley 1994) and will need to be further investigated.

Online data collection conveys major advantages. First, the data quality is positively impacted, as missing, inconsistent or aberrant data can be minimised by adding alerts to users, automating skip and branching logic. These improvements could be even more visible for questions that require some reflection or memory. In our study, the maximal number of aberrant data in the paper questionnaire was observed for starting smoking age. Furthermore, age at the first employment was less consistently reported in older subjects, but agreement was stronger if the web-based version was performed first, suggesting that automatic controls integrated in the web-based version could help to minimise this bias. Second, web-based questionnaire lower the study burden for participants as they can be completed via a user-friendly interface. In our study, filling duration was less perceived as a constraint for the web-based version than for the traditional paper version. Third, Internet introduces a distance between the investigators and the subjects, which may limit self-censorship compared to interviews (Joinson 1999). This hypothesis was confirmed in adults, who were more likely to provide socially desirable responses to personal lifestyle questions and to “agree” to knowledge statements in a telephone survey than in a web survey (Greene et al. 2008). Fourth, adding multimedia support is one other major progress allowed with web-based questionnaires that traditional paper versions will never overcome. They can help to better understand the exact information required and thus improve the quality of data collected. In our sample, every subjects viewing the help video found it useful and 77.3% of subjects have corrected some mistake thanks to the recapitulation form. Almost 95% of the subjects preferred the web-based version, suggesting that Internet could also be a useful tool to keep subjects motivated during long-term follow-up. Fifth, data processing is highly facilitated and reliability is increased by eliminating data entry or scanning of paper forms. In our study, single data entry would have led to at least one mistake for 82% of the subjects (120 among 147). Finally, the avoidance of printing, mailing and data entry in the web-based studies results in considerable reductions in cost and paper waste. Sending this socio-demographic and economic questionnaire to our 500,000 subjects target would have reached the unaffordable cost of 8,235,000 € (US $11,975,502).

Our study includes several limitations. First, paper questionnaires cannot be the gold standard for their Internet-based counterparts since both are prone to bias. Whether web- or paper-based, socio-demographic and economic questionnaires remain both self-administered and are therefore prone to misreporting. Therefore, our study is a comparison and not a validation of Internet-based questionnaires. Second, lassitude of the subjects when completing the second version may also interfere in their answers. However, the lassitude did not differ between the two versions since reliability was similar according to the order of administration (paper or web first). Finally, given that a random selection among all Internet users was impossible, the concordance between our sample and the general Internet user population can be questioned. However, the percentage of subjects with high speed connection (≥512 kbit/s) among those with Internet connection was close to those reported in the general French population (70 vs. 75%, respectively) (http://www.insee.fr/fr/themes/document.asp?ref_id=ip1076&reg_id=0, Internet au quotidien: un Français sur quatre; Insee Première No. 1076, May 2006). Furthermore, given that one-quarter of the subjects declared they were Internet novice or inexperienced, that the mean age was relatively high (62.7 years) and that subjects were used to fill in paper- but not web-based questionnaires, the discrepancy with the general Internet user population, if it exists, would tend to penalise the web-based version and cannot therefore explain the high reliability observed between the two modes.

In conclusion, online surveys can access large and diversified populations with substantial logistic and cost advantages. Our study supports the idea that web-based socio-demographic and economic questionnaires provide information of similar-to-superior quality compared to the traditional paper method. This brings new insights to justify the use of web-based questionnaires as an efficient and appreciated tool for large epidemiological studies.