Skip to main content
Download PDF

Is it worth it? Cost-effectiveness analysis of a commercial physical activity app

BMC Public Health volume 21, Article number: 1950 (2021) Cite this article

Abstract

Background

Government interest in investing in commercial physical activity apps has increased with little evidence of their cost-effectiveness. This is the first study to our knowledge to examine the cost-effectiveness of a commercial physical activity app (Carrot Rewards) despite there being over 100,000 in the major app stores.

Methods

A cost-effectiveness analysis was performed to calculate the incremental cost-effectiveness ratio (ICER) of the app compared to a no-intervention reference scenario using a five-year time horizon. Primary data was collected between 2016 and 2017. Data synthesis, model creation, and statistical analyses were conducted between 2019 and 2020. An age-, sex-, and geography-dependent Markov model was developed assuming a public healthcare payer perspective. A closed cohort (n = 38,452) representing the population reached by Carrot Rewards in two Canadian provinces (British Columbia, Newfoundland & Labrador) at the time of a 12-month prospective study was used. Costs and effects were both discounted at 1.5% and expressed in 2015 Canadian dollars. Subgroup analyses were conducted to compare ICERs between provinces, sexes, age groups, and engagement levels.

Results

Carrot Rewards had an ICER of $11,113 CAD per quality adjusted life year (QALY), well below a $50,000 CAD per QALY willingness-to-pay (WTP) threshold. Subgroup analyses revealed that the app had lower ICERs for British Columbians, females, highly engaged users, and adults aged 35-64 yrs., and was dominant for older adults (65 + yrs). Deterministic sensitivity analyses revealed that the ICER was most influenced by the relative risk of diabetes. Probabilistic sensitivity analyses revealed varying parameter estimates predominantly resulted in ICERs below the WTP threshold.

Conclusions

The Carrot Rewards app was cost-effective, and dominant for older adults. These results provide, for the first time, rigorous health economic evidence for a commercial physical activity app as part of public health programming.

Peer Review reports

Background

Physical inactivity is the most prevalent modifiable chronic disease risk factor with about 85 % of adults worldwide not meeting recommended physical activity guidelines [1,2,3,4]. The case for population-level intervention is increasingly based on the costs of physical inactivity [5, 6]. In response to recent evidence of the societal cost of physical inactivity (globally $53.8 billion USD), [6] the World Health Organization (WHO) set bold physical inactivity reduction targets—15% relative reduction in the global prevalence of insufficient physical activity in adults and adolescents by 2030 [7]. The WHO singled out digital innovation (e.g., mobile health applications, or mHealth apps) as an important component of a broad “systems-based” solution in their Global Action Plan on Physical Activity [7]. Despite their apparent potential, and growing interest amongst governments [8,9,10] and corporations, [11, 12] a small number of commercial physical activity apps have been independently evaluated in peer-reviewed journals (i.e. only about 15 out of more than 100,000 in the major app stores) [13,14,15,16]. Existing evaluations have found mixed results, with some finding app-based interventions increase physical activity, [13, 14] while others report no effect [15, 17]. Most importantly, no cost-effectiveness analysis of any commercial physical activity app has to our knowledge been published to date. In this study, we evaluate the cost-effectiveness of a commercial physical activity app, Carrot Rewards, which was found to increase objectively-measured daily step counts in a 12-month prospective cohort study.

Evidence standards for digital health technologies suggest that in addition to establishing the relevance, acceptability and effectiveness of mHealth apps in increasing physical activity, it is imperative to recognize their economic impact [18,19,20]. Cost-effectiveness studies facilitate public health policy decision making, allowing skilled policy-makers to compare interventions in terms of costs and effects and determine whether funding is justified in fiscally constrained environments [20,21,22]. Despite the proliferation of physical activity apps, which has accelerated since the WHO declared COVID-19 a global pandemic on March 11, 2020, [23] the lack of cost-effectiveness has been cited as a major barrier to policy investment (i.e. government investment in physical activity apps) [18, 20, 24]. A 2017 systematic review of economic evaluations of mHealth solutions in general uncovered 39 studies that largely reported positive economic outcomes (e.g., increase in life years gained, cost savings) [19]. Only nine of the included studies used an app as the mHealth function, and among those none targeted physical activity. The positive outcomes from this review must be interpreted with caution, however, given the lack of rigour in many of the included studies identified by the review authors (e.g., incomplete economic evaluations, short intervention periods, no sensitivity analyses, etc.). The widespread design deficiencies make it difficult to respond to the common criticism that cost-effectiveness is often assumed, without evidence to support it [19].

To address this issue, cost-effectiveness analyses of ‘top tier’ commercial apps should be prioritized given their mass appeal (i.e. the top 2% reporting more than 500,000 monthly active users (MAUs) [25]). Unlike previous cost-effectiveness analyses of physical activity interventions more broadly (outside the mHealth context e.g., mass media campaigns), [26,27,28] analyses should model risk reductions based on objectively-measured (vs. subjective measures) and longer-term (6+ months, the theoretical threshold for behaviour maintenance) [29] changes in physical activity [19, 20, 30]. Age- and sex-specific models should also be used since disease incidence and mortality rates vary widely by demographic group [19, 30, 31]. To further minimize bias, sensitivity analyses are needed to evaluate variables that most influence results [19, 20, 30]. Lastly, while randomized controlled trials (RCTs) are preferred data sources for cost-effectiveness studies, they are notoriously difficult to conduct in fast-paced commercial digital environments. Iribarren et al. (2017) and others suggest that prospective cohort studies, especially longitudinal ones, may also provide high quality data for cost-effectiveness studies of mHealth interventions [19, 20].

Financial health incentive programs continue to be popular with 56% of large U.S. employers (and at least 15% of European employers), [12] for instance, offering rewards worth $946 USD per year to employees for participating in healthy activities [11]. While a concern with financial incentives is that they can be prohibitively costly, [32] technological advances have made tracking and rewarding physical activity easier and more immediate. This, combined with stronger application of behaviour change theory, has driven the cost of rewards down to pennies a day potentially increasing cost-effectiveness [33]. Carrot Rewards was a ‘top tier’ commercial physical activity app available in Canada only (i.e. 1.3+ million downloads, 500,000+ MAUs as of May 2019) [34]. It leveraged gamification elements and concepts from behavioural economics and self-determination theory to reward users with very small ($0.04 CAD) financial incentives (i.e. points redeemable for consumer goods) to walk more [32]. Our objective, therefore, is to conduct a cost-effectiveness analysis of a ‘top tier’ commercial physical activity app that uses financial incentives to drive healthy behaviour.

Methods

A cost-effectiveness analysis was performed to estimate the relative costs and effects of the Carrot Rewards app compared to a reference scenario where no intervention was available. We developed an age-, sex-, and geography-dependent Markov model assuming a public healthcare payer perspective, given Carrot Rewards was initially publicly funded. All costs are in 2015 Canadian dollars, and both costs and quality adjusted life years (QALYs) are discounted at 1.5% per year, as recommended by the Canadian Agency for Drugs and Technologies in Health [35]. A Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist was completed [30]. Additional details regarding the cohort and data sources are in the Additional File 1. Ethical approval for this study was provided by Western University’s Human Research Ethics Board (#113322). This study involved the secondary use of de-identified data. Therefore, the need for informed consent was waived for this secondary data analysis by Western University’s Human Research Ethics Board. All methods were carried out in accordance with the relevant guidelines and regulations. App users were informed of and had to accept the app’s privacy policy describing how de-identified data may be used for reporting purposes and presented in aggregate.

Cohort

The model uses a closed cohort representing the population reached by the intervention at the time of the study (n = 38,452). The cohort from which the data was collected from were users who completed the download and registration of the app during the recruitment period between June 13th to July 10th, 2016. Data from this cohort was collected for 12 months between 2016 and 2017. The target population was female and male youth (13 to 17 years) and adults (18+ years) living in two Canadians provinces: British Columbia and Newfoundland & Labrador (the first two to fund the app for their constituents). App users were classified into four engagement groups: ‘Limited’ (fewer than 12 weeks in which the app was used at least once), ‘Occasional’ (12–23 weeks), ‘Regular’ (24–51 weeks), or ‘Committed’ (52 weeks) (Additional File 2).

Model design

To model improvement in physical activity, we will use daily step counts collected using built-in smartphone accelerometers and reported in a 12-month prospective cohort evaluation of the Carrot Rewards app [31]. Step counts will be linked to chronic disease risk reductions from available databases. The model was developed in 2019 and is presented in Additional File 3. It assumes all cohort members start in a health state free of events, and consists of five chronic diseases with well-established associations with physical inactivity, [36, 37] although we acknowledge new evidence is accumulating that supports the inverse relationship between physical activity and more than 20 other chronic conditions (e.g., depression, bladder cancer, osteoporosis) [38]. Health states comprised: (i) healthy; (ii) ischaemic heart disease (IHD); (iii) stroke; (iv) diabetes mellitus; (v) colorectal cancer; (vi) breast cancer; and (vii) death. We assumed a five-year time horizon and a cycle length of one year. At the end of each cycle, individuals had an annual probability of either remaining in the same health state or transitioning into a different one. Transitions between health states were allowed once per cycle. Individuals in a chronic disease state either remained in the same chronic disease state or transitioned to death. They could not progress backwards to the healthy state and could not have co-morbid conditions.

Data sources

Transition probabilities were based on annual incidence and mortality rates reported in Additional Files 4 and 5, respectively. The data from which these rates were based upon came from Statistics Canada [39] and Canadian Chronic Disease Surveillance System [40] database, and are therefore representative of the actual subgroups. The extent to which Carrot Rewards increased daily step count was drawn from the 12-month cohort study (i.e. 448.8 and 884.6 step per day increase for ‘Regular’ and ‘Committed’ users, respectively) [31] and converted into a standardized energy expenditure metric using the formula from Wu et al. (2000) [41]. Recent meta-analyses of step count monitoring [42] and physical activity incentive interventions [33] report similar daily step count improvements at 6-to-12 months (i.e. 670 and 464–1050 steps/d, respectively). We assumed daily step count increases noted at 12-months would be sustained each year over the five-year time horizon since improvements beyond six months are considered stable [29, 42]. Relative risks are presented in Additional File 6 [43,44,45,46,47,48,49,50,51,52,53,54,55] for ‘Regular’ and ‘Committed’ users only, as the 12-month study showed no improvements in ‘Limited’ and ‘Occasional’ users. Relative risks reflect the improvement in transition probabilities from a healthy state to a diseased state due to physical activity. The studies from which these relative risks were based upon drew from a population of similarly aged male and female youths and adults from similar geographic regions, and we assumed a direct linear relationship between physical activity and risk reduction.

Only direct medical costs were considered (i.e. drugs, physician care, and hospital care). The average annual medical cost for each of the five chronic diseases are presented in Additional File 7 The data from which these costs were based upon came from the Economic Burden of Illness in Canada [56] database, and are therefore representative of the actual subgroups.

The cost of the Carrot Rewards app was based exclusively on the amount spent on loyalty points to reward all users for registering on the app ($0.60), meeting daily step goals ($0.04/day), and completing weekly step challenges ($0.40 if users reached their step goal 10 non-consecutive times in a 2-week period). Intervention costs were paid for by government partners. We assumed individuals had to continue participating in order to maintain the daily step count increase. Additional File 8 presents the estimated annual cost of the intervention based on the 12-month data. For health-related quality of life, utility data as measured by the EQ-5D were obtained from the literature and reported in Additional File 9 [57,58,59,60,61]. Data were retrieved in 2018 and synthesized in 2019.

Sensitivity analyses

To capture uncertainty associated with these parameters, deterministic sensitivity analyses were performed by individually varying each parameter, and a probabilistic sensitivity analysis (Monte Carlo) was performed by varying all parameters concurrently. Transition probabilities between health states followed a beta distribution, risk ratios followed a log-normal distribution, and intervention costs followed a gamma distribution [62]. Analyses were completed in 2020.

Results

Base case

For the no-intervention arm the average discounted QALY amounted to 4.6348 with a cost of $113.16. Implementing Carrot Rewards improved the average QALY by 0.0011 to 4.6359 and increased the cost by $11.86 to $125.02, for an ICER of $11,113.31/QALY. Subgroup analyses are presented in Table 1. The intervention had a lower ICER for: (a) females ($7959.82/QALY) versus males ($15,896.01/QALY), (b) British Columbia ($9945.20/QALY) versus Newfoundland & Labrador ($14,239.54/QALY), (c) ‘Committed’ ($2715.39/QALY) versus ‘Regular’ users ($14,583.77/QALY), and (d) older versus younger users (13–19, $80,376.60/QALY, 20–34: $32,602.45/QALY; 35–49: $4062.39/QALY; 50–64: $7516.79/QALY; 65–79: dominant).

Table 1 Subgroup analyses by province, gender, engagement level, and age
Full size table

Sensitivity analyses

The influence of each parameter is demonstrated in Fig. 1 with larger bars having a greater influence on the variation in the model (also see Additional File 10). The parameter with the largest influence was the relative risk for diabetes with an upper bound of $20,015.08. The findings of the probabilistic sensitivity analysis are shown in Fig. 2. Based on 10,000 simulations, Carrot Rewards was more effective than no-intervention more than 99% of the time and cost more than no-intervention 100% of the time. Figure 3 shows that Carrot Rewards surpassed the no-intervention scenario at a willingness-to-pay (WTP) of $10,386.26.

Fig. 1
figure 1

Tornado diagram for deterministic sensitivity analyses. ICER: incremental cost-effectiveness ratio. High/Low: parameter set to the upper (grey) or lower (blue) bound of the confidence interval

Full size image
Fig. 2
figure 2

Probabilistic sensitivity analyses based on 10,000 Monte Carlo simulations. WTP: willingness-to-pay. Grey ellipse represents 95% confidence intervals

Full size image
Fig. 3
figure 3

Cost-effectiveness acceptability curve

Full size image

Discussion

Main finding

This is the first study to evaluate the cost-effectiveness of a commercial physical activity app despite there being over 100,000 published in major app stores. We found Carrot Rewards was cost-effective over a five-year time horizon relative to an arbitrary WTP threshold of $50,000/QALY ($11,113.31/QALY). For comparison to economic benchmarks, the Canadian Gross Domestic Product per capita is about $60,000 [63]. These results are relevant for countries with publicly-funded healthcare systems (e.g., Canada, U.K., Germany, Australia) but also corporations considering mHealth apps that target employee physical inactivity given the short- (e.g., depression management) and long-term (e.g., type 2 diabetes risk reduction) benefits of physical activity [38]. Until now, policy-makers were not able to compare traditional physical activity interventions (e.g., mass media campaigns, pedometer interventions) with newer mHealth approaches in terms of costs and effects. This study begins to answer the question ‘Are commercial physical activity apps cost-effective?’ and may help policy-makers determine whether funding is justified in light of some of our parameters (e.g., 449–885 steps/day expected intervention effect, intervention cost of $4–$11/year). Specifically, Carrot Rewards produced physical activity increases with incentives that were at least 50 times smaller than what has been used in previous research (e.g., $0.04 vs. $2.00 per day) [33] and corporate settings (e.g., $4–$11 vs. $1247 per year) [11].

Recent evidence suggests that reward size may be less important than other incentive intervention design features (e.g., incentive timing or form) [64]. It has been suggested that manipulating these other features (outlined by Adams et al. [65] and updated by Mitchell et al. [66]) may help reduce the cost of incentives while maintaining or increasing effects [33]. The small incentives used by Carrot Rewards increased physical activity, in part, because they were offered immediately, thereby exploiting the behavioural economic concept of “present bias”, which is the human tendency to prefer payoffs close to the present [67, 68]. This and other theoretically-informed manipulations may appeal to governments and corporations looking to deploy physical activity incentives as efficiently as possible [33]. Others researchers have demonstrated positive effects with physical activity incentives worth $0.09 to $0.75 USD per day when implemented as part of a multicomponent physical activity intervention [69,70,71,72].

Secondary findings

Carrot Rewards had a lower ICER in British Columbia than Newfoundland & Labrador, possibly explained by higher engagement levels in British Columbia. As well, the app’s ICER was two times lower in females than males and was cost-effective for all age groups over 20 years and dominant over 65 years. The larger effects by age are due to higher baseline rates of chronic conditions as age increases, leading to a greater number of chronic conditions prevented. By sex, although there is a larger incremental effect by males, there is also a larger incremental cost by males, this is due to a smaller reduction in costs related to chronic conditions among males and a larger cost associated with the Carrot application. Carrot Rewards’s ICER was five-times lower in users who engaged for 52 weeks versus those who engaged less often. Notably, in this study, ‘Limited’ and ‘Occasional’ app users incurred costs without benefit. Deterministic sensitivity analyses revealed that estimated cost-effectiveness was most influenced by the relative risk of diabetes. The probabilistic sensitivity analysis revealed that varying parameter estimates across a wide range of uncertainty mostly resulted in ICERs below the $50,000 WTP threshold, and a small number of iterations resulted in ICERs above this threshold. Taken together, our results suggest that an mHealth app with incentives may be most cost-effective for working aged (20 to 64 yrs) and older (65 + yrs) females. As well, continued efforts to maximize app engagement (e.g., with regular behavioural science-informed feature upgrades) [15] and minimize reward magnitudes (e.g., by weaning users off daily incentives after 3–4 months) [33] may yield greater cost-effectiveness.

Related studies

Beyond the dearth of cost-effectiveness evaluations of commercial physical activity apps, a few related studies help put our results in context. Cost-effectiveness evaluations of physical activity interventions in general have mostly determined that pedometer-based interventions are most cost-effective in Australia, Belgium and the Netherlands (with ICERs ranging from €11,100/QALY to dominant) [73,74,75]. For example, one pedometer-based intervention reported an estimated cost-savings of €500 per person and 0.11–0.16 QALYs gained (more favorable than reported for Carrot Rewards) [74]. Similarly, workplace physical activity incentive programmes have proved to be cost-effective in the UK (with ICERs ranging from not cost-effective to £2900/QALY) [76, 77]. For example, a cost-effectiveness study of a workplace physical activity incentive programme reported estimated incremental costs of £4100 and 1.2 QALYs gained [77]. The considerable disparity in study designs generally makes it difficult to draw direct comparisons to the present study. For instance, the current mHealth intervention was delivered on a population-scale (vs. similar studies that base predictions on pilot data), [74] used an objectively-measured physical activity outcome (vs. similar studies that used change estimates from separate meta-analyses, or self-report), [73,74,75] and had a younger sample (vs. older samples in similar studies which increases cost-effectiveness) [74, 78] making head-to-head comparisons difficult. We should note that while Carrot Rewards had a relatively small impact on overall QALYs, the impact was greater amongst ‘Committed’ and older users. This is somewhat comparable to previous studies who reported QALY increases of 0.16 and 0.11 for males and females, respectively, but who used a longer time horizon and whose risk reductions were not based on objectively-measured step count increases.58

Limitations and future directions

We made multiple conservative decisions in how our model was structured. First, given that death was a possible outcome, our decision to employ a 5-year time horizon may omit longer term consequences. Extending the horizon may lead to more favorable ICERs. Second, our decision to not include co-morbidities may have also led to less favorable ICERs, as co-morbidities would increase medical costs while decreasing quality of life. Therefore, the positive effects of physical activity delaying disease onset would have produced more favorable results. Finally, we did not include all chronic diseases associated with physical inactivity (e.g., mood and anxiety disorders) [79]. The inclusion of these diseases and their physical activity-related risk reductions into the model would have resulted in more favorable ICERs. This is notable as 27.3% of Carrot Rewards users self-reported a physician diagnosed mood and anxiety disorder after the current study period (unpublished). Therefore, since we assumed all users started in a healthy state, we may have underestimated cost-effectiveness. As we were not able to link participants’ diagnoses with their objectively-measured physical activity, future studies should also consider the health and economic outcomes of similar interventions stratified by health status. We also made some assumptions to convert objectively-measured daily step count increases to risk reductions. First, we assumed a direct linear relationship between physical activity and risk reduction when in fact it is curvilinear with greater risk reduction at lower initial doses (e.g., going from 0 to 30 min as opposed to from 120 to 150 min). This suggests that our model may be underestimating risk reduction and cost-effectiveness given the generally low baseline physical activity levels of our cohort (i.e. 43% accumulated less than 5000 steps per day) [31]. Second, while our intervention was 12 months-long we assumed physical activity increases persisted for five years, as others have done with similar or shorter duration interventions [21, 73,74,75]. Third, while conservative, we assumed daily step count increases occurred at the lower end of moderate intensity [38] which may not have been the case. Future economic evaluations of accelerometer-based interventions should also take differences in physical activity intensity into consideration when estimating effects on risk reduction.

We also took the perspective of a publicly funded healthcare system as rewards were funded by government partners. A societal perspective would have more favorable ICERs. Our model also includes programmatic costs of the financial incentives only. The model does not include company overhead, as company overhead was funded through other (non-government) revenue sources, or development costs as they were viewed as sunk costs—not normally included as they have no impact on the marginal costs and benefits of continuing an intervention [80].

Conclusions

This is the first cost-effectiveness study of a commercially available physical activity app. We have shown that an incentive-based mHealth app targeting physical activity would be cost-effective in two Canadian provinces over a five-year time horizon. As digital health technologies continue to evolve to address emerging and persistent global health issues, it is incumbent on researchers and policy-makers to demand a broader evidence-base that includes health economic impacts to inform public health policy decisions.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

WHO:

World Health Organization

ICER:

Incremental cost-effectiveness ratio

MAU:

Monthly active user

mHealth:

Mobile health

RCT:

Randomized controlled trial

CAD:

Canadian dollars

USD:

U.S. dollars

QALY:

Quality-adjusted life years

CHEERS:

Consolidated Health Economic Evaluation Reporting Standards

IHD:

Ischemic heart disease

EQ-5D:

EuroQol 5D

References

  1. Colley RC, Garriguet D, Janssen I, Craig CL, Clarke J, Tremblay MS. Physical activity of Canadian adults: accelerometer results from the 2007 to 2009 Canadian health measures survey. Heal reports. 2011;22(1):7.

    Google Scholar 

  2. Althoff T, Hicks JL, King AC, Delp SL, Leskovec J. Large-scale physical activity data reveal worldwide activity inequality. Nature. 2017;547(7663):336–9. https://doi.org/10.1038/nature23018.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Bassett DR Jr, Wyatt HR, Thompson H, Peters JC, Hill JO. Pedometer-measured physical activity and health behaviors in United States adults. Med Sci Sports Exerc. 2010;42(10):1819–25. https://doi.org/10.1249/MSS.0b013e3181dc2e54.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Kohl HW 3rd, Craig CL, Lambert EV, Inoue S, Alkandari JR, Leetongin G, et al. The pandemic of physical inactivity: global action for public health. Lancet. 2012;380(9838):294–305. https://doi.org/10.1016/S0140-6736(12)60898-8.

    Article  PubMed  Google Scholar 

  5. Krueger H, Turner D, Krueger J, Ready AE. The economic benefits of risk factor reduction in Canada: tobacco smoking, excess weight and physical inactivity. Can J Public Heal. 2014;105(1):e69–78. https://doi.org/10.17269/cjph.105.4084.

    Article  Google Scholar 

  6. Ding D, Lawson KD, Kolbe-Alexander TL, Finkelstein EA, Katzmarzyk PT, Van Mechelen W, et al. The economic burden of physical inactivity: a global analysis of major non-communicable diseases. Lancet. 2016;388(10051):1311–24. https://doi.org/10.1016/S0140-6736(16)30383-X.

    Article  PubMed  Google Scholar 

  7. World Health Organization (WHO). Global action plan on physical activity 2018–2030: more active people for a healthier world. Geneva: World Health Organization; 2018.

    Google Scholar 

  8. National Healthy Living Platform: “Carrot Rewards” targets lifestyle improvements - Canada.ca [Internet]. [cited 2021 Oct 4]. Available from: https://www.canada.ca/en/news/archive/2015/07/national-healthy-living-platform-carrot-rewards-targets-lifestyle-improvements.html

  9. Smart Applications - Ministry of Health and Prevention - UAE [Internet]. [cited 2021 Oct 4]. Available from: https://www.mohap.gov.ae/en/services/Pages/DownloadApps.aspx

  10. South Africa Digital Health Dashboard | HealthEnabled [Internet]. [cited 2021 Oct 4]. Available from: http://healthenabled.org/wordpress/south-africa-digital-health-dashboard/

  11. Willis Towers Watson. Willis Towers Watson’s 23rd Annual Best Practices in Health Care Employer Survey. 2018.

    Google Scholar 

  12. Mercer. Europe Corporate Wellness Trends: How Benefits Are Shaping the Future of Work. 2019.

    Google Scholar 

  13. Feter N, Dos Santos TS, Caputo EL, da Silva MC. What is the role of smartphones on physical activity promotion? A systematic review and meta-analysis. Int J Public Health. 2019;64(5):679–90. https://doi.org/10.1007/s00038-019-01210-7.

    Article  PubMed  Google Scholar 

  14. Petersen JM, Prichard I, Kemps E. A comparison of physical activity mobile apps with and without existing web-based social networking platforms: systematic review. J Med Internet Res. 2019;21(8):e12687. https://doi.org/10.2196/12687.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Romeo A, Edney S, Plotnikoff R, Curtis R, Ryan J, Sanders I, et al. Can smartphone apps increase physical activity? Systematic review and meta-analysis. J Med Internet Res. 2019;21(3):e12053. https://doi.org/10.2196/12053.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Bondaronek P, Alkhaldi G, Slee A, Hamilton FL, Murray E. Quality of publicly available physical activity apps: review and content analysis. JMIR mHealth uHealth. 2018;6(3):e9069. https://doi.org/10.2196/mhealth.9069.

    Article  Google Scholar 

  17. Schoeppe S, Alley S, Van Lippevelde W, Bray NA, Williams SL, Duncan MJ, et al. Efficacy of interventions that use apps to improve diet, physical activity and sedentary behaviour: a systematic review. Int J Behav Nutr Phys Act. 2016;13(1):1–26. https://doi.org/10.1186/s12966-016-0454-y.

    Article  Google Scholar 

  18. Rahimi K. Digital health and the elusive quest for cost savings. Lancet Digit Heal. 2019;1(3):e108–9. https://doi.org/10.1016/S2589-7500(19)30056-1.

    Article  Google Scholar 

  19. Iribarren SJ, Cato K, Falzon L, Stone PW. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PLoS One. 2017;12(2):e0170581. https://doi.org/10.1371/journal.pone.0170581.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. National Institute for Health and Care Excellence. Evidence Standards Framework for Digital Health Technologies. 2019.

    Google Scholar 

  21. Roux L, Pratt M, Tengs TO, Yore MM, Yanagawa TL, Van Den Bos J, et al. Cost effectiveness of community-based physical activity interventions. Am J Prev Med. 2008;35(6):578–88. https://doi.org/10.1016/j.amepre.2008.06.040.

    Article  PubMed  Google Scholar 

  22. Shah GH, Bogaert K, Yin J, Gould E, Leider JP. Gaps in public health workers’ awareness of emerging public health trends. J Public Heal Manag Pract. 2020;26(5):401–3. https://doi.org/10.1097/PHH.0000000000001049.

    Article  Google Scholar 

  23. Peloton app installs rise 245%, but does it mean anything? - Business of Apps [Internet]. [cited 2021 Oct 4]. Available from: https://www.businessofapps.com/insights/peloton-app-installs-rise-245-but-does-it-mean-anything/

  24. World Health Organization (WHO). mHealth: New horizons for health through mobile technologies. Vol. 3. Global Observatory for eHealth series; 2011.

    Google Scholar 

  25. Research2Guidance. mHealth Economics 2017/2018: Current Status and Future Trends in Mobile Health 2017.

  26. Laine J, Kuvaja-Köllner V, Pietilä E, Koivuneva M, Valtonen H, Kankaanpää E. Cost-effectiveness of population-level physical activity interventions: a systematic review. Am J Health Promot. 2014;29(2):71–80. https://doi.org/10.4278/ajhp.131210-LIT-622.

    Article  PubMed  Google Scholar 

  27. Abu-Omar K, Rütten A, Burlacu I, Schätzlein V, Messing S, Suhrcke M. The cost-effectiveness of physical activity interventions: a systematic review of reviews. Prev Med reports. 2017;8:72–8. https://doi.org/10.1016/j.pmedr.2017.08.006.

    Article  Google Scholar 

  28. Mizdrak A, Telfer K, Direito A, Cobiac LJ, Blakely T, Cleghorn CL, et al. Health gain, cost impacts, and cost-effectiveness of a mass media campaign to promote smartphone apps for physical activity: modeling study. JMIR mHealth uHealth. 2020;8(6):e18014. https://doi.org/10.2196/18014.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot. 1997;12(1):38–48. https://doi.org/10.4278/0890-1171-12.1.38.

    Article  CAS  PubMed  Google Scholar 

  30. Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated health economic evaluation reporting standards (CHEERS) statement. Int J Technol Assess Health Care. 2013;29(2):117–22. https://doi.org/10.1017/S0266462313000160.

    Article  PubMed  Google Scholar 

  31. Mitchell M, Lau E, White L, Faulkner G. Commercial app use linked with sustained physical activity in two Canadian provinces: a 12-month quasi-experimental study. Int J Behav Nutr Phys Act. 2020;17(1):1–9. https://doi.org/10.1186/s12966-020-00926-7.

    Article  Google Scholar 

  32. Rondina R II, Pearson EK, Prapavessis H, White L, Richard S, Mitchell MS. Bright spots, physical activity investments that (almost) worked: carrot rewards app, driving engagement with pennies a day. Br J Sports Med. 2020;54(15):927–9. https://doi.org/10.1136/bjsports-2019-100969.

    Article  Google Scholar 

  33. Mitchell MS, Orstad SL, Biswas A, Oh PI, Jay M, Pakosh MT, et al. Financial incentives for physical activity in adults: systematic review and meta-analysis. Br J Sports Med. 2020;54(21):1259–68. https://doi.org/10.1136/bjsports-2019-100633.

    Article  PubMed  Google Scholar 

  34. Toronto tech company plans to relaunch Carrot Rewards wellness app - The Globe and Mail [Internet]. [cited 2021 Oct 4]. Available from: https://www.theglobeandmail.com/business/technology/article-toronto-tech-company-plans-to-relaunch-carrot-rewards-wellness-app/

  35. Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 4th Edition. Ottawa; 2017.

  36. Janssen I. Health care costs of physical inactivity in Canadian adults. Appl Physiol Nutr Metab. 2012;37(4):803–6. https://doi.org/10.1139/h2012-061.

    Article  PubMed  Google Scholar 

  37. Warburton DER, Charlesworth S, Ivey A, Nettlefold L, Bredin SSD. A systematic review of the evidence for Canada’s physical activity guidelines for adults. Int J Behav Nutr Phys Act. 2010;7(1):1–220. https://doi.org/10.1186/1479-5868-7-39.

    Article  Google Scholar 

  38. Riebe D, Ehrman JK, Liguori G, Magal M. Medicine AC of S. ACSM’s guidelines for exercise testing and prescription. Wolters Kluwer; 2018.

    Google Scholar 

  39. Statistics Canada: Canada’s national statistical agency [Internet]. [cited 2018 Oct 4]. Available from: https://www.statcan.gc.ca/eng/start

  40. Canadian Chronic Disease Surveillance System (CCDSS) [Internet]. [cited 2018 Oct 4]. Available from: https://health-infobase.canada.ca/ccdss/data-tool/

  41. Wu S, Cohen D, Shi Y, Pearson M, Sturm R. Economic analysis of physical activity interventions. Am J Prev Med. 2011;40(2):149–58. https://doi.org/10.1016/j.amepre.2010.10.029.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Chaudhry UAR, Wahlich C, Fortescue R, Cook DG, Knightly R, Harris T. The effects of step-count monitoring interventions on physical activity: systematic review and meta-analysis of community-based randomised controlled trials in adults. Int J Behav Nutr Phys Act. 2020;17(1):1–16. https://doi.org/10.1186/s12966-020-01020-8.

    Article  Google Scholar 

  43. Sundquist K, Qvist J, Johansson SE, Sundquist J. The long-term effect of physical activity on incidence of coronary heart disease: a 12-year follow-up study. Prev Med (Baltim). 2005;41(1):219–25. https://doi.org/10.1016/j.ypmed.2004.09.043.

    Article  Google Scholar 

  44. Mai PL, Sullivan-Halley J, Ursin G, Stram DO, Deapen D, Villaluna D, et al. Physical activity and colon cancer risk among women in the California teachers study. Cancer Epidemiol Biomark Prev. 2007;16(3):517–25. https://doi.org/10.1158/1055-9965.EPI-06-0747.

    Article  Google Scholar 

  45. Friedenreich C, Norat T, Steindorf K, Boutron-Ruault MC, Pischon T, Mazuir M, et al. Physical activity and risk of colon and rectal cancers: the European prospective investigation into Cancer and nutrition. Cancer Epidemiol Biomark Prev. 2006;15(12):2398–407. https://doi.org/10.1158/1055-9965.EPI-06-0595.

    Article  Google Scholar 

  46. Myint PK, Luben RN, Wareham NJ, Welch AA, Bingham SA, Day NE, et al. Combined work and leisure physical activity and risk of stroke in men and women in the European prospective investigation into cancer-Norfolk prospective population study. Neuroepidemiology. 2006;27(3):122–9. https://doi.org/10.1159/000095551.

    Article  PubMed  Google Scholar 

  47. Calling S, bo H, Engström G, Berglund G, Janzon L. Effects of body fatness and physical activity on cardiovascular risk: risk prediction using the bioelectrical impedance method. Scand J Public Health. 2006;34(6):568–75. https://doi.org/10.1080/14034940600595621.

    Article  PubMed  Google Scholar 

  48. Armstrong MEG, Green J, Reeves GK, Beral V, Cairns BJ. Frequent physical activity may not reduce vascular disease risk as much as moderate activity: large prospective study of women in the United Kingdom. Circulation. 2015;131(8):721–9. https://doi.org/10.1161/CIRCULATIONAHA.114.010296.

    Article  CAS  PubMed  Google Scholar 

  49. Folsom AR, Kushi LH, Hong CP. Physical activity and incident diabetes mellitus in postmenopausal women. Am J Public Health. 2000;90(1):134–8. https://doi.org/10.2105/ajph.90.1.134.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Howard RA, Freedman DM, Park Y, Hollenbeck A, Schatzkin A, Leitzmann MF. Physical activity, sedentary behavior, and the risk of colon and rectal cancer in the NIH-AARP diet and health study. Cancer Causes Control. 2008;19(9):939–53. https://doi.org/10.1007/s10552-008-9159-0.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Hu FB, Leitzmann MF, Stampfer MJ, Colditz GA, Willett WC, Rimm EB. Physical activity and television watching in relation to risk for type 2 diabetes mellitus in men. Arch Intern Med. 2001;161(12):1542–8. https://doi.org/10.1001/archinte.161.12.1542.

    Article  CAS  PubMed  Google Scholar 

  52. Rockhill B, Willett WC, Hunter DJ, Manson JE, Hankison SE, Colditz GA. A prospective study of occupational physical activity and breast cancer risk. Cancer Causes Control. 2015;26(12):1779–89. https://doi.org/10.1007/s10552-015-0671-8.

    Article  Google Scholar 

  53. McTiernan A, Kooperberg C, White E, Vogel VG. Recreational physical activity and the risk of breast cancer in postmenopausal women: the women’s health initiative cohort study. Breast Dis. 2004;15(1):25–6. https://doi.org/10.1097/01.OGX.0000115859.71446.89.

    Article  Google Scholar 

  54. Okada K, Hayashi T, Tsumura K, Suematsu C, Endo G, Fujii S. Leisure-time physical activity at weekends and the risk of type 2 diabetes mellitus in Japanese men: the Osaka health survey. Diabet Med. 2000;17(1):53–8. https://doi.org/10.1046/j.1464-5491.2000.00229.x.

    Article  CAS  PubMed  Google Scholar 

  55. Tanasescu M, Leitzmann MF, Rimm EB, Willett WC, Stampfer MJ, Hu FB. Exercise type and intensity in relation to coronary heart disease in men. J Am Med Assoc. 2002;288(16):1994–2000. https://doi.org/10.1001/jama.288.16.1994.

    Article  Google Scholar 

  56. Economic Burden of Illness in Canada - Canada.ca [Internet]. [cited 2018 Oct 4]. Available from: https://www.canada.ca/en/public-health/services/chronic-diseases/chronic-disease-knowledge-development-exchange/economic-burden-illness-canada.html

  57. Golicki D, Niewada M, Buczek J, Karlińska A, Kobayashi A, Janssen MF, et al. Validity of EQ-5D-5L in stroke. Qual Life Res. 2015;24(4):845–50. https://doi.org/10.1007/s11136-014-0834-1.

    Article  PubMed  Google Scholar 

  58. Färkkilä N, Sintonen H, Saarto T, Järvinen H, Hänninen J, Taari K, et al. Health-related quality of life in colorectal cancer. Color Dis. 2013;15(5):e215–22. https://doi.org/10.1111/codi.12143.

    Article  Google Scholar 

  59. Yousefi M, Najafi S, Ghaffari S, Mahboub-Ahari A, Ghaderi H. Comparison of SF-6D and EQ-5D scores in patients with breast cancer. Iran Red Crescent Med J. 2016;18(5):e23556.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Bansback N, Tsuchiya A, Brazier J, Anis A. Canadian valuation of EQ-5D health states: Preliminary value set and considerations for future valuation studies. PLoS One. 2012;7(2):e31115.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  61. Tsiplova K, Pullenayegum E, Cooke T, Xie F. EQ-5D-derived health utilities and minimally important differences for chronic health conditions: 2011 Commonwealth Fund survey of sicker adults in Canada. Qual Life Res. 2016;25(12):3009–16. https://doi.org/10.1007/s11136-016-1336-0.

    Article  PubMed  Google Scholar 

  62. Briggs A, Sculpher M. Claxton K. Oup Oxford: Decision modelling for health economic evaluation; 2006.

    Google Scholar 

  63. GDP per capita (current US$) - Canada | Data [Internet]. [cited 2021 Oct 4]. Available from: https://data.worldbank.org/indicator/NY.GDP.PCAP.CD?locations=CA

  64. Brower J, LaBarge MC, White L, Mitchell MS. Examining responsiveness to an incentive-based Mobile health app: longitudinal observational study. J Med Internet Res. 2020;22(8):e16797. https://doi.org/10.2196/16797.

    Article  PubMed  PubMed Central  Google Scholar 

  65. Adams J, Giles EL, McColl E, Sniehotta FF. Carrots, sticks and health behaviours: a framework for documenting the complexity of financial incentive interventions to change health behaviours. Health Psychol Rev. 2014;8(3):286–95. https://doi.org/10.1080/17437199.2013.848410.

    Article  PubMed  Google Scholar 

  66. Mitchell MS, Goodman JM, Alter DA, Oh PI, Faulkner GEJ. Development of the health incentive program questionnaire (HIP-Q) in a cardiac rehabilitation population. Transl Behav Med. 2015;5(4):443–59. https://doi.org/10.1007/s13142-015-0330-3.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Camerer CF, Loewenstein G. Behavioral economics: past, present, future. In: Advances in Behavioral Economics, Princeton, Princeton University Press Chang, H (2000)‘A Liberal Theory of Social Welfare: Fairness, Utility, and the Pareto Principle’, Yale Law Review. Citeseer; 2003.

    Google Scholar 

  68. Wertenbroch K, Soman D, Chattopadhyay A. On the perceived value of money: the reference dependence of currency numerosity effects. J Consum Res. 2007;34(1):1–10. https://doi.org/10.1086/513041.

    Article  Google Scholar 

  69. Carrera M, Royer H, Stehr M, Sydnor J. Can financial incentives help people trying to establish new habits? Experimental evidence with new gym members. J Health Econ. 2018;58:202–14. https://doi.org/10.1016/j.jhealeco.2018.02.010.

    Article  PubMed  PubMed Central  Google Scholar 

  70. Condliffe S, Işgın E, Fitzgerald B. Get thee to the gym! A field experiment on improving exercise habits. J Behav Exp Econ. 2017;70:23–32. https://doi.org/10.1016/j.socec.2017.07.007.

    Article  Google Scholar 

  71. Rohde KIM, Verbeke W. We like to see you in the gym—a field experiment on financial incentives for short and long term gym attendance. J Econ Behav Organ. 2017;134:388–407. https://doi.org/10.1016/j.jebo.2016.12.012.

    Article  Google Scholar 

  72. Leahey TM, Subak LL, Fava J, Schembri M, Thomas G, Xu X, et al. Benefits of adding small financial incentives or optional group meetings to a web-based statewide obesity initiative. Obesity. 2015;23(1):70–6. https://doi.org/10.1002/oby.20937.

    Article  PubMed  Google Scholar 

  73. Cobiac LJ, Vos T, Barendregt JJ. Cost-effectiveness of interventions to promote physical activity: a modelling study. PLoS Med. 2009;6(7):e1000110. https://doi.org/10.1371/journal.pmed.1000110.

    Article  PubMed  PubMed Central  Google Scholar 

  74. De Smedt D, De Cocker K, Annemans L, De Bourdeaudhuij I, Cardon G. A cost-effectiveness study of the community-based intervention ‘10 000 Steps Ghent.’. Public Health Nutr. 2012;15(3):442–51. https://doi.org/10.1017/S1368980011001716.

    Article  PubMed  Google Scholar 

  75. Over EAB, Wendel-Vos GCW, van den Berg M, Hamberg-van Reenen HH, Tariq L, Hoogenveen RT, et al. Cost-effectiveness of counseling and pedometer use to increase physical activity in the Netherlands: a modeling study. Cost Eff Resour Alloc. 2012;10(1):1–7. https://doi.org/10.1186/1478-7547-10-13.

    Article  Google Scholar 

  76. Hunter RF, Murray JM, Gough A, Tang J, Patterson CC, French DP, et al. Effectiveness and cost-effectiveness of a loyalty scheme for physical activity behaviour change maintenance: results from a cluster randomised controlled trial. Int J Behav Nutr Phys Act. 2018;15(1):1–13. https://doi.org/10.1186/s12966-018-0758-1.

    Article  Google Scholar 

  77. Dallat MAT, Hunter RF, Tully MA, Cairns KJ, Kee F. A lesson in business: cost-effectiveness analysis of a novel financial incentive intervention for increasing physical activity in the workplace. BMC Public Health. 2013;13(1):1–9. https://doi.org/10.1186/1471-2458-13-953.

    Article  Google Scholar 

  78. Frew EJ, Bhatti M, Win K, Sitch A, Lyon A, Pallan M, et al. Cost-effectiveness of a community-based physical activity programme for adults (be active) in the UK: an economic analysis within a natural experiment. Br J Sports Med. 2014;48(3):207–12. https://doi.org/10.1136/bjsports-2012-091202.

    Article  PubMed  Google Scholar 

  79. Schuch FB, Vancampfort D, Firth J, Rosenbaum S, Ward PB, Silva ES, et al. Physical activity and incident depression: a meta-analysis of prospective cohort studies. Am J Psychiatry. 2018;175(7):631–48. https://doi.org/10.1176/appi.ajp.2018.17111194.

    Article  PubMed  Google Scholar 

  80. Cellini SR, Kee JE. Cost-effectiveness and cost-benefit analysis. Handb Pract Progr Eval. 2010;3:493-560.

Download references

Acknowledgements

We acknowledge Guy Faulkner and Erica Lau for providing the point estimates that were previously published in our other studies and used as inputs for the model of the current paper.

Funding

The Public Health Agency of Canada (PHAC) Multi-Sectoral Partnership Approach to Healthy Living and Chronic Disease Prevention grant, as well as the Baycrest Hospital’s Soupcoff Family Fellowship, the Mitacs Elevate program, the Sandra Rotman Centre for Health Sector Strategy Fellowship, and the Social Sciences and Humanities Research Council Partnership Grant. The funding bodies did not contribute to any aspect of the study.

Author information

Authors and Affiliations

  1. Rotman School of Management, University of Toronto, Toronto, ON, Canada

    Renante Rondina

  2. Schulich School of Medicine and Dentistry, Western University, London, ON, Canada

    Michael Hong & Sisira Sarma

  3. Faculty of Health Sciences, Western University, London, ON, Canada

    Marc Mitchell

Authors
  1. Renante Rondina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sisira Sarma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marc Mitchell
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All co-authors meet the authorship criteria outlined in the Proceedings of the National Academy of Sciences of the United States of America. All author have read and approved the final manuscript.

Corresponding authors

Correspondence to Renante Rondina or Marc Mitchell.

Ethics declarations

Ethics approval and consent to participate

Ethical approval for this study was provided by Western University’s Human Research Ethics Board (#113322). This study involved the secondary use of de-identified data. Therefore, the need for informed consent was waived for this secondary data analysis by Western University’s Human Research Ethics Board. All methods were carried out in accordance with relevant guidelines and regulations. App users were informed of and had to accept the app’s privacy policy describing how de-identified data may be used for reporting purposes and presented in aggregate.

Consent for publication

Not applicable.

Competing interests

MM received consulting fees from Carrot Insights Inc., makers of the Carrot Rewards app, from 2015 to 2018 as well as travel re-imbursement in January and March 2019. MM had stock options in the company as well but these are now void since Carrot Insights Inc. went out of business in June 2019. RR was employed as a research intern by Carrot Insights Inc. from March 2018 to July 2018, and completed a Mitacs postdoctoral fellowship at the company from February 2019 to June 2019. MH and SS declare that they have no conflicts of interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Detailed description of methods, cohort, and data sources.

Additional file 2.

Cohort composition by age, gender, geography, and engagement level.

Additional file 3.

Illustration of Markov model design: in this state-transition diagram all of the seven health states are inserted. Each arrow is linked with a certain transition probability. Circles represent possible health states. The following states can be distinguished: healthy, diabetes, colorectal cancer, breast cancer, (ischemic) heart disease, stroke, and death.

Additional file 4.

Age-, gender-, and geography-dependent transition probabilities by chronic disease.

Additional file 5.

Age-, gender-, and disease-dependent mortality rates by health state.

Additional file 6.

Relative risk by engagement level, age group, and gender.

Additional file 7.

Age-, gender-, and geography dependent average annual medical costs by chronic disease.

Additional file 8.

Age-, gender-, and geography-dependent average annual intervention costs by engagement level.

Additional file 9.

Age-, gender-, and geography dependent average annual medical costs by chronic disease.

Additional file 10.

Lower and upper bounds for each parameter in the deterministic sensitivity analysis.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rondina, R., Hong, M., Sarma, S. et al. Is it worth it? Cost-effectiveness analysis of a commercial physical activity app. BMC Public Health 21, 1950 (2021). https://doi.org/10.1186/s12889-021-11988-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12889-021-11988-y

Keywords

BMC Public Health

ISSN: 1471-2458

Contact us