Assessing the reach and effectiveness of mHealth: evidence from a reproductive health program for adolescent girls in Ghana

Study setting

The study was conducted in the urban area of Accra, Ghana [1]. In a survey from 2015 in Ghana, 31% of respondents aged 14–18 years and 71% of respondents aged 19–25 years owned a mobile phone, while 77% of those aged 14–18 and 91% of those aged 19–25 used a mobile phone in the past 4 weeks [20].

Despite enacting a National Adolescent Reproductive Health Policy in 1996, the SRH of Ghanaian adolescents remains a significant challenge. According to the 2014 Demographic and Health Survey, by age 19, 36% of women have given birth or are pregnant [22]. Among adolescent mothers under 20 years of age, 58% had not planned on having the pregnancy at that time. Use of contraception among adolescents is low: among sexually active unmarried adolescents aged 15–19, less than a third use a modern method of contraception [22].

Though adolescents are generally aware of HIV and pregnancy risks, SRH knowledge tends to be limited and mostly superficial. Results of a 2004 national survey of Ghanaian adolescents found that while more than 95% of girls aged 15–19 knew of at least one method of modern contraception, only 67% were aware that a woman is more likely to get pregnant on certain days than others. Only 28% knew both previous items and were able to reject two common misconceptions (that a woman cannot get pregnant the first time she has sex or if she has sex standing up) [23]. Moreover, widespread misconceptions about the correct way to use contraception and fears about long-term side effects on future fertility add to the barriers to modern contraception use among adolescents [24–26].

Procedures

We used secondary data from a recent randomized trial that provided SRH information by text message (also known as short message service, or SMS) to adolescent girls in Ghana. The original trial, which is described in further detail in Rokicki et al. (2016), found relatively large and persistent increases in SRH knowledge among adolescents participating in the interactive mHealth program [5].

The sampling frame for the study was provided by the 2012–2013 Ghana Education Service Register of Secondary Schools in Greater Accra. We randomized 38 schools to the interactive mHealth intervention (n = 12), a simplified unidirectional messaging intervention (n = 12), and the control group (n = 14). After randomization, we found 3 schools to be ineligible and 1 refused to participate due to time constraints [5]. In this paper, we focus only on the interactive mHealth intervention arm and the control arm. The analytic sample consisted of 10 schools assigned to the intervention (N = 205) and 12 schools assigned to control (N = 293).

We blocked randomization by school category (a measure of quality designated by the Ghana Education Service) and by whether the school had a home economics class. We chose classes within schools based on maximizing the number of eligible girls; if a home economics class was offered in the school it was chosen because it was always a large class of nearly all female students. Home economics was not offered at all schools so stratifying ensured even distribution across arms.

The inclusion criteria for schools was senior high day schools in the Greater Accra region; boarding schools were excluded. Within schools, sampling was restricted to female students, aged 14–24 years, in one class in the second year of senior high school. Participants used their own mobile phones or could use a family member’s phone; no phones were provided.

Participants gave written consent; those aged younger than 18 years obtained parental consent. Institutional Review Board (IRB) approval was granted by Harvard University [#FWA00004837] as well as locally by the Ghana Health Service (GHS-ERC:05/09/13). The study design was registered on ClinicalTrials.gov (NCT02031575).

We visited schools at baseline, immediately after the intervention was completed (3-month follow-up), and 1 year later (15-month follow-up). At each visit, participants completed a self-administered questionnaire. Participants’ demographic information was completed at baseline. Reproductive health knowledge was assessed at each time-point. Information on sexual behavior and pregnancies was collected only at the 15-month follow-up.

The mHealth platform

The mHealth platform was designed as an interactive mobile phone quiz game in which participants could win airtime (i.e. mobile phone credit that can be used for making calls or sending texts) for texting correct answers to SRH questions. To design the message content, we first conducted focus groups with adolescents to understand their most prevalent SRH concerns, followed by a round table discussion with the Health Promotion Unit at Ghana Health Service who approved appropriateness of topics and finalized wording.

For a period of 12 weeks, participants were sent one multiple-choice quiz question about SRH each week via text message to which they were invited to respond free of charge. These messages focused on pregnancy prevention and contained information on topics of reproductive anatomy, pregnancy, sexually transmitted infections (STIs), and contraception including male and female condoms, birth control pills, and emergency contraception. Upon responding, participants immediately received a confirmatory text message informing them whether they answered correctly, the correct answer, and additional information. Participants were sent up to 2 reminder messages if they did not respond; those who had not responded by the end of the week were sent a text message with the correct answer and the additional information at the end of the week. Participants in the same school were encouraged to discuss messages with each other. Participants were told that correct answers were rewarded: for every 2 correct responses, participants were sent 1 GHS (US$0.38) of airtime credit at the end of the week.

Over the same 12-week intervention period, the control group participants were sent one message each week with information about malaria. The content of all messages is shown in Table 4 in Appendix. Participants in the control group were interviewed at baseline, 3 months, and 15 months using the same procedures as participants in the intervention group. All messages were sent in English, the official language of instruction in all secondary schools in the country, using secure servers via Telerivet service; non-delivered messages were re-sent.

Measures

To measure program engagement, we used two separate variables. First, we measured the total number of times the respondent replied to the weekly text-message quiz questions (maximum of 12), from data extracted from the mobile phone records. Second, we measured self-reported message exposure. At the 3-month follow up, respondents were asked “How often did you receive messages from [the program]” [More than once a week/ About once a week/ About once a month/ Less than once a month]. We created an indicator for having received messages at least once a week.

To evaluate reproductive health knowledge, participants completed a true-or-false test consisting of 24 items (see Table 5 in Appendix for details). Items on the test were adapted from the Guttmacher Institute’s 2009 National Survey of Reproductive and Contraceptive Knowledge for the setting of Ghana [27]. Knowledge scores were calculated as the percentage of items answered correctly; we then calculated knowledge z-scores by subtracting from each score the overall mean at baseline and dividing by the standard deviation.

In calculating knowledge scores, “don’t know” answers and missing values were treated as incorrect answers. The percentage missing for each item was low and is shown in Table 5 in Appendix. At the 15-month follow-up, data collection was done using tablet computers so there were no missing values in those scores. For the baseline and the 3-month follow-up, which were done on paper questionnaires, we re-calculated knowledge scores such that items with missing values were excluded from the calculation (that is, the knowledge score was calculated as the percentage correct of the total number of items answered). The correlation coefficients between treating missing values as incorrect and excluding missing values from the calculation was 0.92 at baseline and 0.99 at the first follow-up.

Self-reported pregnancy was assessed at the 15-month follow-up with the question “In the past year, have you been pregnant?” [Yes/ No/ I don’t want to answer].

Our explanatory variables were selected ex-ante on the basis of theory and prior evidence of risk and protective factors for adolescent SRH [13]. We identified these as: 1) adolescents from households of low socioeconomic status (defined as both parents completed only primary school or less), 2) adolescents who were sexually active at baseline, 3) adolescents who have a larger than average SRH knowledge deficit (earned a baseline knowledge z-score of less than 0), and 4) adolescents who have low parental support (do not strongly agree or agree a little bit with the statement “I feel comfortable talking to my parents about condoms and contraception” [Strongly agree, Agree a little bit, Neither agree nor disagree, Disagree a little bit, Strongly disagree]).

Statistical analysis

Our analysis is divided in three parts. First, we evaluated characteristics associated with engagement with the interactive mHealth program. We used a Poisson regression model to examine the association between number of responses to the text-messsage quiz questions as our outcome and parental education, sexual experience, parental support, and SRH knowledge as explanatory variables. We also verified similarity of results to a negative binomial model to account for over-dispersion (Table 6 in Appendix). Additionally, we used logistic regressions to examine the associations of these characteristics with a binary variable for any response to the text-messages, as well as with a binary variable measuring participation as whether the participant self-reported to have received messages at least once a week during the course of the program.

Next, to evaluate the impact of engaging with the program on reproductive health knowledge, we used linear regression models of knowledge z-score at both 3 and 15 months as a function of level of engagement (number of responses) of the intervention group. We used quantile-quantile plots to assess normality of the residuals.

Finally, we assessed variation in program impact across target subgroups. We used linear regression models stratified by subgroup to estimate the impact of the intervention on knowledge z-score. We then used models with an interaction term for intervention group and subgroup indicator to test equality of the coefficients. For these interaction tests, marginal effects were derived from the interaction model, and then compared using linear hypothesis tests.

Due to the small cell sizes in subgroups, we used exact logistic regression models, again stratified by subgroup, to estimate the impact of the intervention on self-reported pregnancy in the past year.

Missing values and item response refusals were included in the analyses as separate categories; results for these categories are shown in Tables 6 and 7 in Appendix. We adjusted all models for blocking variables, which were category of school (higher quality public, lower quality public, and private) and an indicator for the presence of home economics class) and we clustered standard errors at the school level to correct for within-school correlation of outcomes [28, 29]. We used Stata v14 for all analyses [30].

Differences in program engagement by target subgroups

Association between engagement and knowledge

Figure 1 shows the difference in average knowledge z-score between intervention and control groups as a function of the intervention group's engagement with the program (as measured by the number of responses). Higher levels of engagement were associated with higher knowledge scores both at 3 months (linear slope estimate 0.11, 95%CI 0.08 to 0.14) and at 15 months (linear slope estimate 0.07, 95%CI 0.02 to 0.13).

Difference in program impact on high risk groups

Figure 2 shows the difference in knowledge z-score for the intervention compared to the control, stratified by target subgroups. The mHealth program was effective at significantly increasing knowledge for every subgroup at both 3 and 15 months. We found no heterogeneity in the effect of the intervention across any subgroup population (Table 7 in Appendix).