Can we trust measures of healthcare utilization from household surveys?
BMC Public Health volume 13, Article number: 853 (2013)
The research community relies heavily on measures of healthcare utilization from household surveys to understand health seeking choices and to evaluate interventions in developing countries. Such measures are known to suffer from recall problems but there is limited evidence of whether the method of data collection affects evaluation findings. We compared the results of a randomized trial of free healthcare using utilization data from two sources.
Data are from a study in Ghana, in which 2,194 households containing 2,592 children under 5 y old were randomized into a prepayment scheme providing free primary and some referral care, or to a control group whose families paid user fees for healthcare. Data on morbidity and health seeking behaviour were collected using a standard household survey administered at endline and a pictorial diary given to households over a six month period, collected at monthly intervals.
Self-reported measures of morbidity and healthcare utilization were substantially lower in the household survey than the pictorial diary when the recall period was over a month. Introducing free healthcare had a positive effect on primary care visits based on the pictorial diary and a non-significant negative effect according to the household survey. Using any clinic visit in the past month as the outcome, the difference in the effect of free care between the two data collection methods was 3.6 percentage points (p = 0.078).
The findings raise methodological concerns about measures of healthcare utilization from household surveys, particularly in the evaluation of health financing interventions.
Household surveys are frequently used to understand health seeking behaviour, particularly in developing countries where routine information systems may be unreliable for lack of investment and capacity. They are considered the standard tool for measuring healthcare utilization at the population level. A typical survey first asks household members whether they have been ill over a defined period and if so, whether they sought care and from whom. Recall periods vary between studies but standard practice is to extend the recall period to no more than one month in the case of outpatient care, and no more than a year in the case of inpatient care.
Measures of healthcare use from household surveys are widely used by the research community – to understand care seeking choices and to evaluate health programmes. Studies of the effect of health financing policies in developing countries rely heavily on such measures. Prominent examples in the recent literature include a randomized assessment of the Mexican universal health insurance programme  and various evaluations of the cooperative medical scheme in rural China [2, 3]. Utilization measures from household surveys also provide the basis for conducting benefit incidence analyses in health [4, 5].
To what extent can we trust measures of healthcare utilization from household surveys? There are several reasons to be cautious. First, individuals may struggle to recall episodes of illness and visits to healthcare providers. This failure to recall may vary systematically across population groups, thereby generating biases that have important implications for interpretation of the data. It is well documented, for example, that poorer individuals are less likely to report illness even though they experience greater morbidity . A recent study in India experimentally assigned households to surveys with weekly and monthly recall periods, finding that the recall period had a large effect on reported morbidity, doctor visits and self-medication . The impacts were particularly large for poorer households and those with a high burden of disease. However, when recall periods are shorter bias may be less of a problem. A study conducted in multiple developing countries showed that using a 7-day rather than a 2-day recall generated little bias in self-reports of illness . Second, recall may vary according to the financing arrangements in place. If an individual has to pay out-of-pocket for care, he or she may be more likely to remember a doctor visit than someone who is covered by health insurance, particularly when it is costly. Such detection bias is why whenever feasible randomized controlled trials conceal assignment .
Much of the literature on problems with survey methods has addressed the issue of recall, but few studies have explored methods of data collection. In this paper we focus on the latter using data from rural Ghana. We first compare household survey measures of morbidity and healthcare utilization against our benchmark measure taken from a pictorial diary designed specifically for the study. We then take the novel step of comparing the results of a randomized trial of removing direct payments for healthcare using both sets of utilization measures. Our findings raise a number of concerns that have important methodological implications for our understanding of health reporting and the evaluation of policies on utilization of healthcare in resource-poor settings.
We used data collected for the purposes of a randomized experiment of removing direct payments for healthcare, in which one of the authors (EA) was the Principal Investigator. The study was conducted in Dangme West, a poor rural district in Southern Ghana with an estimated 2004 mid-year population of 115,000. The study provided free primary and some referral healthcare to households randomly assigned to the intervention group by paying for them to enrol into an existing prepayment health insurance scheme in May 2004. Households in the control group continued to pay a fee-for-service at public health facilities in accordance with the national policy at the time. Malaria was the leading cause of morbidity and mortality in children less than five years of age in the study district.
Households with at least one child aged 6 to 59 months and not already enrolled in the prepayment health insurance scheme were eligible to participate in the study. The sample frame consisted of approximately 8,700 households with children under five years of age living in the study area. Of the 2,332 households (with 2,757 children of eligible age) that were randomly selected from the district database to participate in the study, 138 households (with 165 children) had already enrolled voluntarily into the prepayment health insurance scheme by the time the registration window had closed. They were excluded from the main study but retained as an observational arm. The remaining 2,194 households with 2,592 children were randomly assigned to treatment and control groups.
At the baseline household survey in May 2004 a total of 2,151 households with 2,524 children were found and interviewed. In the final household survey, carried out at the end of the malaria transmission season in December 2004, 1,981 households with 2,321 children were successfully follow-up. In total, we have endline data for 2,319 children. Further details of the study – its design, the main outcomes and the findings – are available elsewhere [10, 11].
Measures of morbidity and healthcare utilization
We have two sources of data on healthcare utilization. The first was a panel household survey in which caretakers of the study children were interviewed before and after the introduction of the free care intervention. Questions relevant to this study included, “When was the last time your child fell ill?” and “When your child was last ill where did you seek care?” In contrast to most other household surveys, the questionnaire did not impose a specific recall period on the respondent. Rather, the first question was left open and responses were coded according to time categories. Based on responses to the two questions we construct binary indicators of the most recent illness, primary care clinic visit, and informal care visit for the following recall periods: past month, and past year. Informal care refers to traditional healers, chemists, or drug peddlars.
The second data collection method was a pictorial diary, supplied to households after the start of the intervention. This second source of data provides our benchmark or ‘gold standard’ measure of healthcare utilization. A diary is a research tool that requires respondents to make regular records of their daily activities and experiences. They are thought to be particularly useful in situations where there are likely to be difficulties in recalling past events, experiences or behaviours . Diaries have been used as a data collection tool in many different fields of research. Some of these areas include nutrition , sexual behaviour [14–17], alcohol consumption [18, 19] as well as lifestyle diaries . Diaries are increasingly being mainstreamed into Living Standards Measurement Surveys conducted in numerous countries around the world as a means to capture information on household spending and consumption. The advantage of diaries is that the data are recorded as the events occur and therefore less likely to be affected by recall bias [21, 22]. When used for a long period diaries can also provide a picture of variations in the behaviour under study. Conversely, the diary method may lead to a reduction in participant retention and compliance if the time and effort investment required is too high [21, 23, 24].
A pictorial diary was developed specifically for this study to collect information from households on episodes of illness and healthcare seeking over a six month follow-up period (Figures 1 and 2). A key consideration to collecting reliable information was the fact that a substantial proportion of the mothers were illiterate. It was a two-part diary, the first of which depicted signs of common childhood illness (e.g. fever, vomiting, and convulsion). The pictures were adapted from the existing and well-known Road to Health Chart for children used widely in the country. The second part was designed by a local artist working closely with the Principal Investigator and depicted various healthcare seeking possibilities (e.g. primary care clinic, hospital, chemical seller, traditional healer and home). Both were thoroughly pre-tested locally for comprehension and feasibility.
The pictorial diaries were collected every month by trained fieldworkers. Caregivers were requested to make a mark underneath the relevant picture each time to indicate what illness the child suffered from and what healthcare (if any) was sought. Feedback from the study participants suggested that, anecdotally at least, the diaries were popular.
From the diary data we develop the following morbidity measures: annual number of illness episodes per person; any illness in the past month; and any illness over the six month diary data collection period. Corresponding measures are constructed for primary care clinic visits and informal care visits. For morbidity and utilization in the past month, we use data from the November pictorial diary to correspond with the month of recall in the household survey. In doing so we ensure that these measures are directly comparable between the two data sources, ruling out any influence of seasonality.
There are several parts to the analysis. First, we compare mean values of morbidity and healthcare utilization from the two data sources, using the control group only. To assess the extent of under-reporting, we focus on measures of morbidity and utilization in the past month since these indicators are similarly defined. We also do the same for measures with the longer reference period (six months for the pictorial diary data and one year for the household data) while recognising that the comparison is not entirely valid.
Second, we estimate the effect of removing direct payments for healthcare on the use of primary healthcare and informal healthcare from the two sources of data. We use ordinary least squares and report both unadjusted estimates and estimates adjusted for age, education of mother, number of children in the household, household wealth, gender, distance to the nearest health facility, religion and ethnicity. In all regressions, we cluster the standard errors at the household level to deal with correlation between children in the same household and to account for the fact that treatment assignment was done on household basis.
Finally, we examine in a single regression framework whether the method of data collection influences the effect of the free care intervention on utilization of health services. We stack the utilization data from the two sources and run a regression in which we include a free care dummy, a household data collection dummy, and an interaction between the two. The coefficient on the interaction identifies the difference in the effect of free care between the two data sources. As before, we include controls and cluster the standard errors at the household levela.
Table 1 provides descriptive statistics of the morbidity and utilization measures. According to the pictorial diary data, 41% of children had at least one episode of illness in the past month, compared with 46% in the household survey data. When the reference period is extended, the difference between the two sources of data becomes large. The pictorial diary data show that 93% of children had at least one episode of illness over the six month period of data collection. By contrast, the proportion of children with any illness in the past year based on the household survey data was 60% despite the fact that the reference period is twice as long. These data suggest there was severe under-reporting of illness in the household survey. At least one third of morbidity (33%) – and probably more if the reference periods were identical – was not captured by the household survey.
The proportion of children who sought healthcare in the past month is reasonably similar across the two sources of data in the case of both clinic visits (18% pictorial diary vs. 15% household survey) and informal care visits (16% pictorial diary vs. 21% household survey). When the reference period is lengthened to six months/ one year, the method of data collection appears important in the measurement of any clinic visit (63% pictorial diary vs. 22% household survey) and informal care visits (67% pictorial diary vs. 27% household survey).
Table 2 reports the effect of removing direct payments for healthcare across the different measures of utilization. The intervention had a positive effect on primary care visits according to the pictorial diary data (Table 2, Panel A). The unadjusted estimates indicate it increased the number of visits by 0.30 per year (unadjusted rate ratio of 12 percent) and the probability of any visit in the past six months by 3.7 percentage points (unadjusted rate ratio of 6 percent). According to the household survey data, the intervention had no statistically significant effect on clinic use (Table 2, Panel B). The point estimates are negative suggesting that, if anything, the intervention reduced primary care use. The point estimates for informal care utilization are negative irrespective of which data collection method was used. Only in the case of informal care visits from the pictorial diary is the estimate significant, with a negative effect of 0.27 per year (unadjusted rate ratio of 9 percent).
Table 3 shows the extent to which the method of data collection affects the estimated impact of the free care intervention on healthcare utilization by reporting the coefficient on the interaction between free care and the data collection method. With respect to clinic visits, the data collection method appears to matter. Depending on the reference period, the effect of free care on use of clinic visits is 3.6 percentage points to 5.4 percentage points smaller with the household data. In the case of informal care visits, there are no differences in the effect of free care with respect to the data collection method.
Using pictorial diary data as our benchmark, we addressed the question of whether household survey data on health seeking behaviour provide reliable measures of healthcare use. We found substantial under-reporting of morbidity and healthcare utilization when recall periods were extended beyond one month and qualitatively different conclusions as to the effect of removing user fees on primary care use. On a practical level the study demonstrated the feasibility of implementing pictorial diaries in a rural population where a sizeable proportion of respondents were illiterate. The caregivers did not appear to have any difficulty in using the pictorial diaries as a means of reporting health events.
The severe under-reporting of morbidity was consistent with a study in India examining how different recall periods affect reporting . Since morbidity acts as the standard screening question prior to asking about health seeking behaviour in household surveys, this result inevitably implies substantial under-reporting of healthcare use. The findings were also consistent with those of a study in the US on alcohol consumption and sexual activity which found that participants more often underreported their sexual activities and condom use during retrospective studies as compared to when diaries were used .
Evidence on how the survey method alters the conclusions of the randomized trial of removing direct payments for healthcare is the most novel aspect of the paper. The change in the direction of effect between data sources is both intriguing and worrisome, given the ubiquitous use of household survey data in the evaluation of health policies. We may have viewed the findings with more scepticism had they been based on an observational study. But the randomized design and the fact that the treatment effects are negative for both of the two household survey measures of primary care utilization lend credibility to the findings.
We can only speculate as to the reason for this finding but one hypothesis is that the recall of families in the intervention group deteriorated precisely because they no longer had to pay for healthcare, making the event less salient and more easily forgotten, thereby dampening the true effect of the interventionb. If so it would provide an example of how detection bias can emerge passively through the influence of an intervention on the recall of self-reported behaviours. Because the intervention did not affect the price of care at informal providers the salience and recollection of informal care seeking events should not have been affected by the intervention. Indeed our findings regarding the effect of the intervention on informal care visits were consistent across the two methods of data collection, providing further support to this hypothesis.
The study had a number of limitations that should be noted. First, although the contribution of the study is methodological, it is based on empirical findings from one district in Ghana. However, there is good reason to believe recall problems are universal. Extensive evidence from the US showing that around 20 percent of outpatient events are under-reported when the recall period extends beyond two to three days  has been confirmed by studies in developing countries [26, 27]. Second, owing to data limitations we were only able to construct one indicator of utilization that was standardised across the two data collection methods. A greater number of comparable measures of utilization would have strengthened the analysis. Finally, our method does not allow recall effects to be separated from data collection methods, although it seems likely the channel through which the data collection method altered the effect of free care is via recall.
The findings raise methodological concerns about measures of healthcare utilization from household surveys, particularly in the evaluation of health financing interventions. Further research is needed into the problems of bias when collecting utilization data in household surveys. Such data are widely used by health researchers to understand health seeking behaviour, in benefit incidence analysis and in the evaluation of interventions and policies. The implications of the study’s findings are therefore potentially far reaching. It may be the case that the trend towards using household diaries of income and expenditure should be extended to the collection of data on healthcare utilization.
aWe ran regressions in which we included child fixed-effects instead of the demographic controls. The results are very similar to those produced in Table 3. The findings are also qualitatively the same if we use a logit model instead of OLS.
bThere is a suggestion in the paper by Das et al. (2012) that much of what is underreported in household surveys when the recall period is relatively long are the less severe illnesses associated with smaller household expenditures .
King G, Gakidou E, Imai K, Lakin J, Moore RT, Nall C, Ravishankar N, Vargas M, Tellez-Rojo MM, Avila JE, Avila MH, Llamas HH: Public policy for the poor? A randomised assessment of the Mexican universal health insurance programme. Lancet. 2009, 373: 1447-1454. 10.1016/S0140-6736(09)60239-7.
Babiarz KS, Miller G, Yi H, Zhang L, Rozelle S: New evidence on the impact of China's New Rural Cooperative Medical Scheme and its implications for rural primary healthcare: multivariate difference-in-difference analysis. BMJ. 2010, 341: c5617-10.1136/bmj.c5617.
Wagstaff A, Lindelow M, Jun G, Ling X, Juncheng Q: Extending health insurance to the rural population: an impact evaluation of China's new cooperative medical scheme. J Health Econ. 2009, 28: 1-19. 10.1016/j.jhealeco.2008.10.007.
Mills A, Ataguba JE, Akazili J, Borghi J, Garshong B, Makawia S, Mtei G, Harris B, Macha J, Meheus F, McIntyre D: Equity in financing and use of health care in Ghana, South Africa, and Tanzania: implications for paths to universal coverage. Lancet. 2012, 380: 126-133. 10.1016/S0140-6736(12)60357-2.
O'Donnell O, van Doorslaer E, Wagstaff A, Lindelow M: Analysing health equity using household survey data: a guide to techniques and their implementation. 2008, Washington DC: World Bank
Sen A: Health: perception versus observation. BMJ. 2002, 324: 860-861. 10.1136/bmj.324.7342.860.
Das J, Hammer J, Sanchez-Paramo C: The impact of recall periods on reported morbidity and health seeking behavior. J Dev Econ. 2012, 98: 76-88. 10.1016/j.jdeveco.2011.07.001.
Arnold BF, Galiani S, Ram PK, Hubbard AE, Briceno B, Gertler PJ, Colford JM: Optimal recall period for caregiver-reported illness in risk factor and intervention studies: a multicountry study. Am J Epidemiol. 2013, 177: 361-370. 10.1093/aje/kws281.
Higgins J, Altman D, Sterne J: Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions Version 510. Edited by: Higgins J, Green S. 2011, Available from http://www.cochrane-handbook.org. The Cochrane Collaboration
Ansah EK, Narh-Bana S, Asiamah S, Dzordzordzi V, Biantey K, Dickson K, Gyapong JO, Koram KA, Greenwood BM, Mills A, Whitty CJ: Effect of removing direct payment for health care on utilisation and health outcomes in Ghanaian children: a randomised controlled trial. PLoS Med. 2009, 6 (1): 48-58.
Powell-Jackson T, Hanson K, Whitty CJ, Ansah EK: Who Benefits from Removing User Fees for Health Care? Evidence from a Randomised Experiment in Ghana. 2012
Wiseman V, Conteh L, Matovu F: Using diaries to collect data in resource-poor settings: questions on design and implementation. Health Policy Plan. 2005, 20: 394-404. 10.1093/heapol/czi042.
Witschi C: Short-term dietary recall and recording methods. Nutritional Epidemiology. Edited by: Willett W. 1990, New York: Oxford University Press
Coxon A: Diaries and sexual behavior: The use of sexual diaries as method and substance in researching gay men’s response to HIV/AIDS. Challenge and innovation and methodological advances in social research on HIV/AIDS: Social aspects of AIDS. Edited by: Boulton M. 1994, London: Taylor & Francis
Gillmore M, Gaylord J, Hartway J, Hoppe MJ, Morrison DM, Leigh BC, Rainey DT: Daily data collection of sexual and other health-related behaviors. J Sex Res. 2001, 38: 35-42. 10.1080/00224490109552068.
Hornsby PP, Wilcox AJ: Validity of questionnaire information on frequency of coitus. Am J Epidemiol. 1989, 130: 94-99.
McAuliffe TL, DiFranceisco W, Reed BR: Effects of question format and collection mode on the accuracy of retrospective surveys of health risk behavior: a comparison with daily sexual activity diaries. Health Psychol. 2007, 26: 60-67.
Leigh BC, Gillmore MR, Morrison DM: Comparison of diary and retrospective measures for recording alcohol consumption and sexual activity. J Clin Epidemiol. 1998, 51: 119-127. 10.1016/S0895-4356(97)00262-X.
Poikolainen K, Karkkainen P: Diary gives more accurate information about alcohol consumption than questionnaire. Drug Alcohol Depend. 1993, 11: 209-216.
Coxon AP, Coxon NH, Weatherburn P, Hunt AJ, Hickson F, Davies PM, McManus TJ: Sex role separation in sexual diaries of homosexual men. AIDS. 1993, 7: 877-882. 10.1097/00002030-199306000-00018.
Verbrugge LM: Health diaries. Med Care. 1980, 18: 73-95.
Whitty C, Jones RJ: A comparison of prospective and retrospective diary methods of assessing alcohol use among university undergraduates. J Public Health Med. 1992, 14: 264-270.
Wheeler L, Reis H: Self-recording of everyday life events: origins, types and uses. J Pers. 1991, 59: 339-354. 10.1111/j.1467-6494.1991.tb00252.x.
Stone AA, Kessler RC, Haythornthwaite JA: Measuring daily events and experiences: decisions for the researcher. J Pers. 1991, 59: 575-607. 10.1111/j.1467-6494.1991.tb00260.x.
Cannell C, Marguiz K, Laurent A: A Summary of Studies of Interviewing Methodology. 1977, Washington, D.C.: Government Printing Office
OECD: Private health expenditure - report on the conclusions of the project: 11th meeting of health accounts experts and correspondents for health expenditure data. 2009, Paris: Organisation for Economic Co-operation and Development
Ross DA, Vaughan JP: Health interview surveys in developing countries: a methodological review. Stud Fam Plann. 1986, 17: 78-94. 10.2307/1967068.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/13/853/prepub
We wish to acknowledge the staff of the Dangme West District Health Directorate and members of participating communities in the Dangme West District. We thank Professor Christopher Whitty who provided comments on an earlier draft of the paper. There was no funding for the preparation of this manuscript. The Bill & Melinda Gates Foundation funded the randomized controlled trial of free healthcare on which this paper is based. During the original trial, the PI also received support from the Gates Malaria Partnership of the London School of Hygiene and Tropical Medicine for the period of her PhD. The funder did not have any role in the study design, data collection and analysis, decision to publish, or preparation of this manuscript.
The authors declare that they have no competing interests.
EA designed the evaluation, carried out the data collection, contributed to the analysis and drafting of the manuscript. TPJ carried out the analysis and drafting of the manuscript. Both authors read and approved the final manuscript.
About this article
Cite this article
Ansah, E.K., Powell-Jackson, T. Can we trust measures of healthcare utilization from household surveys?. BMC Public Health 13, 853 (2013). https://doi.org/10.1186/1471-2458-13-853