The intervention was associated with a 5% point increase in households that reported toilet use by all members aged 5 years or above. In the control arm, about 85% of households were consistent toilet users. In this light, a 5% increase could be interpreted as a relevant effect given that only around 15% of the target households (households with an existing toilet) may have been inconsistent or non-users prior to intervention. It could therefore be argued that a third of those households who were in a position to improve toilet use behaviour, did so as a consequence of the campaign. However, our alternative tool to measure toilet use, the physical activity questionnaire, which we assume to be less likely to be influenced by responder bias, showed no evidence for an effect. We believe the effect estimate of the primary outcome is likely to be subject to over-reporting of toilet use in the intervention arm. This is supported by the comparison of exposed and unexposed in the intervention arm, which resulted in a major difference in the primary outcome measured by the main tool, but not in the physical activity tool (summarised in Fig. 3). The results therefore suggest that the 5 Star Toilet campaign in this rural Indian setting with high pre-existing toilet coverage and probable high levels of use did not further increase toilet use.
The campaign was delivered by trained facilitators and follow-up in the community was done through village volunteers. The word ‘smart’ was translated as saru (good), saras (nice) and sunder (beautiful) by the facilitators while delivering the campaign. In this manner the campaign attempted to mainstream the 5 Star Toilet concept by placing it in the context of other desirable, modern things in people’s lives.
There are a number of possible reasons for the failure of the intervention to achieve major changes in toilet use behaviour. First the intervention may have been ill conceived, second it may have failed to reach enough of the target audience with enough intensity to effect measurable change, third it may have been delivered to a population who were already convinced of the need to use toilets, leaving only a small number of potential users who could not be persuaded.
Process evaluation data suggests that, for those who participated, the programme was well received. Through interviews and focus group discussions with participants exposed to the campaign and regional government representatives, we found that intervention components surprised the participants and were different from what people may have experienced before, in government or NGO-led initiatives. Through discussions with participants, the most commonly reported motives for toilet improvement included comfort, convenience, affiliation, status and honour related to women’s safety. It thus appeared that our theory of change for how the intervention would lead to toilet use was supported, at least for those who received the intervention.
However, the exposure of the target population to the intervention was low. Only about 10–15% of the intervention households showed evidence of exposure to the intervention. This low exposure was insufficient to change the study population’s perceptions around toilet ownership and other relevant sanitation-related factors at village level. Small positive changes in toilet features and proxy markers of current use were observed (see Table 4) but statistical support for these changes was low and could have occurred by chance.
The low exposure to the 5 Star Toilet campaign may have been due to the fact that clusters in the study area were geographically spread out and various socio-economic strata and caste groups/religious communities lived in different segments of the village. This challenged the campaign facilitators as within the short timeframe available for the day event, it was difficult to reach out to each and every household. The main occupation of people in the study clusters was agricultural, managing livestock and diamond polishing, and many had also migrated to the nearby cities of Surat and Ahmadabad. The absence of householders made it difficult for the campaign facilitators to identify eligible households and to recruit participants for the intervention. It is also plausible that the non-users of toilets were a particularly hard-to contact and hard-to-convince group, since many of those around them had already adopted the practice.
On the whole, intervention intensity was temporarily high in the locations where activities occurred but geographically too scattered and too infrequent to achieve a high exposure at population level. The digital elements of the interventions were well received by the population, but were found to be no substitute for the sustained efforts on the ground likely to be required to achieve behaviour change.
The study findings are in line with other water/sanitation/hygiene-related behaviour change campaigns, such as those targeting handwashing behaviour where small interventions have shown success [17], while larger campaigns at scale have failed to produce major effects [23,24,25]. Delivering behaviour change interventions at scale remains a challenge. Three other trials conducted under the same initiative alongside the present study in other parts of India also failed to achieve relevant changes in toilet use, even though these were carried out in different settings with lower toilet coverage and possibly lower baseline toilet use [20].
Rates of usage of toilets, at 84–88%, by a variety of reported measures, were higher in this study than we initially expected, based on small scale surveys prior to the intervention. One solution to the problem of intervening in a population who in the majority did not need to change behaviour might have been to find a way to target only the approximately 15% of people or households who were not using their latrines consistently. From the programme perspective, intervention efficiency will be reduced if it mainly consists of activities performed at the community level such as public events and road shows. Intervention resources are then wasted on a majority of people attending such events who have no need to change their behaviour. On the other hand, identifying target households within a given community may not be easy without in-depth knowledge from inside the community and serial household visits to increase intervention exposure in those who could benefit from it most. Approaches to identify households not using toilets need to be conducted in a way that avoids stigmatising households based on income, caste and other status-related characteristics. Thorough formative research taking into account knowledge from earlier programmes will be needed to guide the decision on whether to favour a community-level or more targeted approach for a given intervention in a particular setting.
Limitations of the present study include the use of self-reported behaviour to measure the primary outcome, imbalances in some socio-economic variables across study arms, the low coverage of the intervention and the short time frame from randomisation to intervention delivery and outcome assessment.
The study relied on self- or proxy-reported toilet use as the primary outcome, which is likely to lead to over-reporting of socially desirable behaviours. This method was used across all four studies in this programme to ensure standardised reporting of the primary outcome. In the setting of a randomised trial testing the effect of an intervention on socially desirable behaviours (here: toilet use), there is an additional risk of differential reporting behaviour between intervention and control arm. Study participants in the intervention arm who have just been exposed to an intervention may be more prone to over-reporting toilet use than participants in the control arm, for whom the survey may simply appear as just another household survey, unlinked to an intervention. Higher over-reporting of toilet use in the intervention would cause a spurious effect of the intervention on toilet use.
We tried to explore the potential for differential over-reporting influencing the study results by employing a newly developed tool to measure toilet use and open defecation – the physical activity tool. Here, going for open defecation was one of many questionnaire items related to different physical activities throughout the day, alongside other questions related to chronic non-communicable diseases including dietary pattern. This tool found a 4.4% points lower toilet use among all study participants (individual level), and there was no evidence for an increase in toilet use among the intervention households. These findings are compatible with the presence of over-reporting in the primary outcome, and suggest that the observed effect of a 5% percentage point increase in toilet use may be due to differential over-reporting. This is supported by the conspicuous difference in reported toilet use in the intervention arm between those reporting to have heard of the campaign and those that had not. The effect of exposure on reported toilet use clearly depended on the questionnaire (Fig. 3). The main questionnaire showed higher reported toilet use among those in the intervention arm directly exposed to the campaign compared to the unexposed. By contrast, the physical activity tool showed no difference in this comparison. In our view, this is evidence for campaign exposure changing reporting behaviour but not toilet use. The difference between the two tools in the control arm was not great (84% vs 81%, Table 3), suggesting that in the absence of an intervention, and if households are visited only once, over-reporting may be limited.
Ahead of the study we suspected over-reporting to occur even in the control arm if households are visited repeatedly before and after the intervention. We tried to reduce this potential for over-reporting by not repeating questions related to sanitation and toilet use in the same households at baseline and at follow up. Households undergoing these questions at baseline were discarded from further study. This strategy appears to have been successful. Toilet use by all household members at baseline (overall 85%, 87% in the control arm, 83% in the intervention arm) was similar to toilet use by all household members observed in the control arm at follow up (84%), suggesting that the trial procedures did not influence reporting behaviour. These findings further suggest that administration of the physical activity tool, which was done about 7 to 10 days before the endline tool, did not influence responses of the endline tool, possibly by successfully camouflaging the purpose of the physical activity survey as a health and lifestyle survey. The lack of increase in reported toilet use from baseline to follow up is in contrast to findings from the other three trials which were part of this initiative. These trials employed similar study and intervention designs, targeting the same health behaviour (consistent use of already built toilets by all household members). They all used a similar tool to measure the primary outcome of reported toilet use behaviour at individual level in each household. However, these trials revisited the same households at baseline and follow up, whereas we removed all households in the baseline toilet use study from further study. Unlike in our study, these trials found a marked increase in reported toilet use in the control arm (Fig. 4, data extracted from [20]). One could argue that it may have been less straightforward for households in our study to link the purpose of the questionnaire to the intervention compared to studies repeating the same questionnaire in the same people with an intervention in between. However, the strong increase in reported toilet use found in the control arm in the other three trials may also have been due to an increase in state- or district-level sanitation activities immediately following the baseline, thus contaminating the trial sites [20].
The lack of baseline toilet use data in the households included in the endline survey meant that we could not adjust the effect estimates for any imbalances in the primary outcome, or use such data for restricted randomisation to achieve balance. Some endline imbalances were observed in variables associated with the primary outcome (asset index and male education). Adjusting for these variables attenuated the observed effect sizes, but did not fundamentally change the interpretation of the results. On the whole we believe that minimising over-reporting and bias by not revisiting the same households before and after an intervention is more important than achieving a high degree of baseline balance across arms with regard to the study outcome. Bias is difficult to address analytically whereas imbalances are due to a chance process which can be adjusted for (at least to some extent) and interpreted in the light of confidence intervals and the results of other studies (in meta-analysis).
Finally, for programmatic reasons, the trial had to be conducted in a short time frame. Less than 12 months were available for baseline study, randomisation, intervention delivery, outcome assessment and reporting of results. Due to budget constraints, intervention activities in each village had to be reduced to 2 days in total. In retrospect, we doubt whether interventions requiring households to make changes to their toilets and to motivate household members to change their toilet use behaviours can result in success within such a short timeframe and with such limited resources.