The aim of this study was to compare the ITC Netherlands web and telephone surveys on demographic and smoking related variables to assess differences in data quality. Data quality differences were present, but they were small and not consistently favourable for either web or telephone interviewing. Cooperation rates were high for both the web and telephone survey (78%), which can be explained by the use of a well respected market research company and the use of reimbursements.
In our study we found relatively large coverage error differences. For example, 31% of respondents with internet access had a low educational level, while 71% of respondents without internet access had a low educational level. However, in absolute terms, coverage error differences are not very large in the Netherlands, since there are not many people without internet access (14% of the population) or without a fixed line telephone (9% of the population) . As internet access keeps increasing and fixed line telephone access keeps decreasing, as was the case in the last several years [17, 31, 32], the degree of coverage error in web surveys is expected to decrease while the degree of coverage error in telephone surveys is expected to increase. Our web and telephone samples both showed coverage error differences with respect to marital status, educational level, and age, but not with respect to gender. Differences between telephone respondents with and without internet access were somewhat larger than differences between web respondents with and without fixed line telephone.
Non-response combined with coverage error differences were found for both the web and telephone survey with respect to marital status and educational level, but not with respect to gender. The largest difference was found between the telephone survey and the Dutch population of smokers with respect to educational level. The telephone survey contained 12% less lower educated respondents than the Dutch population of smokers. In order to be able to generalise the findings of these surveys to the Dutch population of smokers, weighting the data to this population seems crucial.
Measurement error differences were found on three of the six smoking related variables. Although these differences were significant, they were only minor: the mean differences in the use of answering categories between the web and telephone survey was 4%. Web respondents used the "don't know" options more often than telephone respondents, as is in accordance with what was found in other studies [2, 11, 12]. This occurs primarily because web respondents see the "don't know" option on the screen whereas telephone interviewers do not read it aloud. Another possibility is that web respondents feel less pressure to give an answer when they do not know the answer to the question . Furthermore, telephone respondents were more negative about smoking and had more self efficacy for quitting than web respondents. These differences might be caused by more socially desirable responding with telephone than with web interviewing [2, 12, 16]. Also, telephone respondents answered more often that they were not planning to quit smoking, while web respondents answered more often that they were planning to quit smoking sometime in the future but not within 6 months. This difference might be caused by more balanced answering with web interviewing, due to less perceived time pressure . According to another study with ITC data from the United States, Canada, the United Kingdom, and Australia, more highly educated smokers have more self efficacy for quitting . This was also found with the ITC Netherlands web survey, but not with the telephone survey, suggesting that the web survey results may be more valid than the telephone survey results.
In this study, we also found some potential disadvantages of web interviewing. Smokers with internet access had a higher educational level than smokers without internet access. Since we used a quota sample with among others quotas for educational level for the web survey, this difference did not result in a higher educated web sample. Therefore, we recommend using quotas for educational level when using web interviewing. Another potential disadvantage of web interviewing is the relatively high number of "don't know" answers. In our web survey, the "don't know" category was visible on screen in a grey font and preceded by an extra space to prevent greater use of the "don't know" option. Answering "don't know" was still relatively high with the question about intention to quit smoking among web respondents (4%). This can be solved by recoding "don't know" answers as "no intention to quit smoking", which is already common practice in studies with one interviewing mode [29, 33] and can reduce the differences between a web and telephone sample in mixed mode studies.
Of course, factors other than survey error play a role when choosing an interviewing mode for a particular study. Web surveys are less costly than telephone surveys, and allow for lengthier questionnaires, a shorter fieldwork period, and the inclusion of visual stimuli [1–3]. Budget constraints can be a powerful motivator to choose web interviewing, since the differences in costs are huge. In our study, the costs were €15 (US$22) for fieldwork costs and reimbursements per web respondent and €62 (US$90) per telephone respondent. Some studies suggest that this difference will increase in time, because telephone surveys become increasingly more expensive [1, 34].
Although our study shows that using a mixed mode approach can threaten comparability of results, it can also have advantages. A mixed mode approach can reduce coverage error, when people without access to one mode have access to another mode [1, 3, 6]. Also, using a mixed mode approach in multi-country studies with one mode within each country but different modes across countries can reduce coverage error when one country has high internet access and low telephone access and another country has high telephone access and low internet access [1, 3, 6]. Furthermore, non-response error can be reduced by using a mixed mode approach in which respondents can choose their interviewing mode [3, 6]. Finally, measurement error can be reduced by using a self-administered interviewing mode (for example web interviewing) for answering the sensitive questions that are part of an interviewer administered survey (for example telephone interviewing) [3, 6].
A limitation of our study was that we could not compare the age distributions of our surveys with the Dutch population of smokers. This was due to a deliberate overrepresentation of young smokers in the web sample, for the purpose of another study. This was unfortunate, since age is an important factor in smoking behaviour [23, 24]. Fortunately, we could compare the three samples on education, which is perhaps an even more important factor in smoking behaviour. Since the web and telephone sample both contained more respondents who were not married than the Dutch population of smokers, it is expected that they also contained more younger respondents.
Another limitation was that we used internet access at home as an indicator of the possibility to participate in web surveys. However, internet access at school, work, and at other public places can also give the possibility to participate in web surveys. Our study may therefore have overestimated the problem of coverage error in web interviews.
The Netherlands is a country where 86% of the population has internet access, where 88% of the internet users have a broadband connection , and where web surveying is common practice. Therefore, the results from this study may not be generalisable to countries with lower levels of internet access and less experience with web surveying. However, with increasing internet access and the increasing use of web surveys worldwide, the results from this study may well apply to other countries in the near future.