- Open Access
Is it worth it? Cost-effectiveness analysis of a commercial physical activity app
BMC Public Health volume 21, Article number: 1950 (2021)
Government interest in investing in commercial physical activity apps has increased with little evidence of their cost-effectiveness. This is the first study to our knowledge to examine the cost-effectiveness of a commercial physical activity app (Carrot Rewards) despite there being over 100,000 in the major app stores.
A cost-effectiveness analysis was performed to calculate the incremental cost-effectiveness ratio (ICER) of the app compared to a no-intervention reference scenario using a five-year time horizon. Primary data was collected between 2016 and 2017. Data synthesis, model creation, and statistical analyses were conducted between 2019 and 2020. An age-, sex-, and geography-dependent Markov model was developed assuming a public healthcare payer perspective. A closed cohort (n = 38,452) representing the population reached by Carrot Rewards in two Canadian provinces (British Columbia, Newfoundland & Labrador) at the time of a 12-month prospective study was used. Costs and effects were both discounted at 1.5% and expressed in 2015 Canadian dollars. Subgroup analyses were conducted to compare ICERs between provinces, sexes, age groups, and engagement levels.
Carrot Rewards had an ICER of $11,113 CAD per quality adjusted life year (QALY), well below a $50,000 CAD per QALY willingness-to-pay (WTP) threshold. Subgroup analyses revealed that the app had lower ICERs for British Columbians, females, highly engaged users, and adults aged 35-64 yrs., and was dominant for older adults (65 + yrs). Deterministic sensitivity analyses revealed that the ICER was most influenced by the relative risk of diabetes. Probabilistic sensitivity analyses revealed varying parameter estimates predominantly resulted in ICERs below the WTP threshold.
The Carrot Rewards app was cost-effective, and dominant for older adults. These results provide, for the first time, rigorous health economic evidence for a commercial physical activity app as part of public health programming.
Physical inactivity is the most prevalent modifiable chronic disease risk factor with about 85 % of adults worldwide not meeting recommended physical activity guidelines [1,2,3,4]. The case for population-level intervention is increasingly based on the costs of physical inactivity [5, 6]. In response to recent evidence of the societal cost of physical inactivity (globally $53.8 billion USD),  the World Health Organization (WHO) set bold physical inactivity reduction targets—15% relative reduction in the global prevalence of insufficient physical activity in adults and adolescents by 2030 . The WHO singled out digital innovation (e.g., mobile health applications, or mHealth apps) as an important component of a broad “systems-based” solution in their Global Action Plan on Physical Activity . Despite their apparent potential, and growing interest amongst governments [8,9,10] and corporations, [11, 12] a small number of commercial physical activity apps have been independently evaluated in peer-reviewed journals (i.e. only about 15 out of more than 100,000 in the major app stores) [13,14,15,16]. Existing evaluations have found mixed results, with some finding app-based interventions increase physical activity, [13, 14] while others report no effect [15, 17]. Most importantly, no cost-effectiveness analysis of any commercial physical activity app has to our knowledge been published to date. In this study, we evaluate the cost-effectiveness of a commercial physical activity app, Carrot Rewards, which was found to increase objectively-measured daily step counts in a 12-month prospective cohort study.
Evidence standards for digital health technologies suggest that in addition to establishing the relevance, acceptability and effectiveness of mHealth apps in increasing physical activity, it is imperative to recognize their economic impact [18,19,20]. Cost-effectiveness studies facilitate public health policy decision making, allowing skilled policy-makers to compare interventions in terms of costs and effects and determine whether funding is justified in fiscally constrained environments [20,21,22]. Despite the proliferation of physical activity apps, which has accelerated since the WHO declared COVID-19 a global pandemic on March 11, 2020,  the lack of cost-effectiveness has been cited as a major barrier to policy investment (i.e. government investment in physical activity apps) [18, 20, 24]. A 2017 systematic review of economic evaluations of mHealth solutions in general uncovered 39 studies that largely reported positive economic outcomes (e.g., increase in life years gained, cost savings) . Only nine of the included studies used an app as the mHealth function, and among those none targeted physical activity. The positive outcomes from this review must be interpreted with caution, however, given the lack of rigour in many of the included studies identified by the review authors (e.g., incomplete economic evaluations, short intervention periods, no sensitivity analyses, etc.). The widespread design deficiencies make it difficult to respond to the common criticism that cost-effectiveness is often assumed, without evidence to support it .
To address this issue, cost-effectiveness analyses of ‘top tier’ commercial apps should be prioritized given their mass appeal (i.e. the top 2% reporting more than 500,000 monthly active users (MAUs) ). Unlike previous cost-effectiveness analyses of physical activity interventions more broadly (outside the mHealth context e.g., mass media campaigns), [26,27,28] analyses should model risk reductions based on objectively-measured (vs. subjective measures) and longer-term (6+ months, the theoretical threshold for behaviour maintenance)  changes in physical activity [19, 20, 30]. Age- and sex-specific models should also be used since disease incidence and mortality rates vary widely by demographic group [19, 30, 31]. To further minimize bias, sensitivity analyses are needed to evaluate variables that most influence results [19, 20, 30]. Lastly, while randomized controlled trials (RCTs) are preferred data sources for cost-effectiveness studies, they are notoriously difficult to conduct in fast-paced commercial digital environments. Iribarren et al. (2017) and others suggest that prospective cohort studies, especially longitudinal ones, may also provide high quality data for cost-effectiveness studies of mHealth interventions [19, 20].
Financial health incentive programs continue to be popular with 56% of large U.S. employers (and at least 15% of European employers),  for instance, offering rewards worth $946 USD per year to employees for participating in healthy activities . While a concern with financial incentives is that they can be prohibitively costly,  technological advances have made tracking and rewarding physical activity easier and more immediate. This, combined with stronger application of behaviour change theory, has driven the cost of rewards down to pennies a day potentially increasing cost-effectiveness . Carrot Rewards was a ‘top tier’ commercial physical activity app available in Canada only (i.e. 1.3+ million downloads, 500,000+ MAUs as of May 2019) . It leveraged gamification elements and concepts from behavioural economics and self-determination theory to reward users with very small ($0.04 CAD) financial incentives (i.e. points redeemable for consumer goods) to walk more . Our objective, therefore, is to conduct a cost-effectiveness analysis of a ‘top tier’ commercial physical activity app that uses financial incentives to drive healthy behaviour.
The model uses a closed cohort representing the population reached by the intervention at the time of the study (n = 38,452). The cohort from which the data was collected from were users who completed the download and registration of the app during the recruitment period between June 13th to July 10th, 2016. Data from this cohort was collected for 12 months between 2016 and 2017. The target population was female and male youth (13 to 17 years) and adults (18+ years) living in two Canadians provinces: British Columbia and Newfoundland & Labrador (the first two to fund the app for their constituents). App users were classified into four engagement groups: ‘Limited’ (fewer than 12 weeks in which the app was used at least once), ‘Occasional’ (12–23 weeks), ‘Regular’ (24–51 weeks), or ‘Committed’ (52 weeks) (Additional File 2).
To model improvement in physical activity, we will use daily step counts collected using built-in smartphone accelerometers and reported in a 12-month prospective cohort evaluation of the Carrot Rewards app . Step counts will be linked to chronic disease risk reductions from available databases. The model was developed in 2019 and is presented in Additional File 3. It assumes all cohort members start in a health state free of events, and consists of five chronic diseases with well-established associations with physical inactivity, [36, 37] although we acknowledge new evidence is accumulating that supports the inverse relationship between physical activity and more than 20 other chronic conditions (e.g., depression, bladder cancer, osteoporosis) . Health states comprised: (i) healthy; (ii) ischaemic heart disease (IHD); (iii) stroke; (iv) diabetes mellitus; (v) colorectal cancer; (vi) breast cancer; and (vii) death. We assumed a five-year time horizon and a cycle length of one year. At the end of each cycle, individuals had an annual probability of either remaining in the same health state or transitioning into a different one. Transitions between health states were allowed once per cycle. Individuals in a chronic disease state either remained in the same chronic disease state or transitioned to death. They could not progress backwards to the healthy state and could not have co-morbid conditions.
Transition probabilities were based on annual incidence and mortality rates reported in Additional Files 4 and 5, respectively. The data from which these rates were based upon came from Statistics Canada  and Canadian Chronic Disease Surveillance System  database, and are therefore representative of the actual subgroups. The extent to which Carrot Rewards increased daily step count was drawn from the 12-month cohort study (i.e. 448.8 and 884.6 step per day increase for ‘Regular’ and ‘Committed’ users, respectively)  and converted into a standardized energy expenditure metric using the formula from Wu et al. (2000) . Recent meta-analyses of step count monitoring  and physical activity incentive interventions  report similar daily step count improvements at 6-to-12 months (i.e. 670 and 464–1050 steps/d, respectively). We assumed daily step count increases noted at 12-months would be sustained each year over the five-year time horizon since improvements beyond six months are considered stable [29, 42]. Relative risks are presented in Additional File 6 [43,44,45,46,47,48,49,50,51,52,53,54,55] for ‘Regular’ and ‘Committed’ users only, as the 12-month study showed no improvements in ‘Limited’ and ‘Occasional’ users. Relative risks reflect the improvement in transition probabilities from a healthy state to a diseased state due to physical activity. The studies from which these relative risks were based upon drew from a population of similarly aged male and female youths and adults from similar geographic regions, and we assumed a direct linear relationship between physical activity and risk reduction.
Only direct medical costs were considered (i.e. drugs, physician care, and hospital care). The average annual medical cost for each of the five chronic diseases are presented in Additional File 7 The data from which these costs were based upon came from the Economic Burden of Illness in Canada  database, and are therefore representative of the actual subgroups.
The cost of the Carrot Rewards app was based exclusively on the amount spent on loyalty points to reward all users for registering on the app ($0.60), meeting daily step goals ($0.04/day), and completing weekly step challenges ($0.40 if users reached their step goal 10 non-consecutive times in a 2-week period). Intervention costs were paid for by government partners. We assumed individuals had to continue participating in order to maintain the daily step count increase. Additional File 8 presents the estimated annual cost of the intervention based on the 12-month data. For health-related quality of life, utility data as measured by the EQ-5D were obtained from the literature and reported in Additional File 9 [57,58,59,60,61]. Data were retrieved in 2018 and synthesized in 2019.
To capture uncertainty associated with these parameters, deterministic sensitivity analyses were performed by individually varying each parameter, and a probabilistic sensitivity analysis (Monte Carlo) was performed by varying all parameters concurrently. Transition probabilities between health states followed a beta distribution, risk ratios followed a log-normal distribution, and intervention costs followed a gamma distribution . Analyses were completed in 2020.
For the no-intervention arm the average discounted QALY amounted to 4.6348 with a cost of $113.16. Implementing Carrot Rewards improved the average QALY by 0.0011 to 4.6359 and increased the cost by $11.86 to $125.02, for an ICER of $11,113.31/QALY. Subgroup analyses are presented in Table 1. The intervention had a lower ICER for: (a) females ($7959.82/QALY) versus males ($15,896.01/QALY), (b) British Columbia ($9945.20/QALY) versus Newfoundland & Labrador ($14,239.54/QALY), (c) ‘Committed’ ($2715.39/QALY) versus ‘Regular’ users ($14,583.77/QALY), and (d) older versus younger users (13–19, $80,376.60/QALY, 20–34: $32,602.45/QALY; 35–49: $4062.39/QALY; 50–64: $7516.79/QALY; 65–79: dominant).
The influence of each parameter is demonstrated in Fig. 1 with larger bars having a greater influence on the variation in the model (also see Additional File 10). The parameter with the largest influence was the relative risk for diabetes with an upper bound of $20,015.08. The findings of the probabilistic sensitivity analysis are shown in Fig. 2. Based on 10,000 simulations, Carrot Rewards was more effective than no-intervention more than 99% of the time and cost more than no-intervention 100% of the time. Figure 3 shows that Carrot Rewards surpassed the no-intervention scenario at a willingness-to-pay (WTP) of $10,386.26.
This is the first study to evaluate the cost-effectiveness of a commercial physical activity app despite there being over 100,000 published in major app stores. We found Carrot Rewards was cost-effective over a five-year time horizon relative to an arbitrary WTP threshold of $50,000/QALY ($11,113.31/QALY). For comparison to economic benchmarks, the Canadian Gross Domestic Product per capita is about $60,000 . These results are relevant for countries with publicly-funded healthcare systems (e.g., Canada, U.K., Germany, Australia) but also corporations considering mHealth apps that target employee physical inactivity given the short- (e.g., depression management) and long-term (e.g., type 2 diabetes risk reduction) benefits of physical activity . Until now, policy-makers were not able to compare traditional physical activity interventions (e.g., mass media campaigns, pedometer interventions) with newer mHealth approaches in terms of costs and effects. This study begins to answer the question ‘Are commercial physical activity apps cost-effective?’ and may help policy-makers determine whether funding is justified in light of some of our parameters (e.g., 449–885 steps/day expected intervention effect, intervention cost of $4–$11/year). Specifically, Carrot Rewards produced physical activity increases with incentives that were at least 50 times smaller than what has been used in previous research (e.g., $0.04 vs. $2.00 per day)  and corporate settings (e.g., $4–$11 vs. $1247 per year) .
Recent evidence suggests that reward size may be less important than other incentive intervention design features (e.g., incentive timing or form) . It has been suggested that manipulating these other features (outlined by Adams et al.  and updated by Mitchell et al. ) may help reduce the cost of incentives while maintaining or increasing effects . The small incentives used by Carrot Rewards increased physical activity, in part, because they were offered immediately, thereby exploiting the behavioural economic concept of “present bias”, which is the human tendency to prefer payoffs close to the present [67, 68]. This and other theoretically-informed manipulations may appeal to governments and corporations looking to deploy physical activity incentives as efficiently as possible . Others researchers have demonstrated positive effects with physical activity incentives worth $0.09 to $0.75 USD per day when implemented as part of a multicomponent physical activity intervention [69,70,71,72].
Carrot Rewards had a lower ICER in British Columbia than Newfoundland & Labrador, possibly explained by higher engagement levels in British Columbia. As well, the app’s ICER was two times lower in females than males and was cost-effective for all age groups over 20 years and dominant over 65 years. The larger effects by age are due to higher baseline rates of chronic conditions as age increases, leading to a greater number of chronic conditions prevented. By sex, although there is a larger incremental effect by males, there is also a larger incremental cost by males, this is due to a smaller reduction in costs related to chronic conditions among males and a larger cost associated with the Carrot application. Carrot Rewards’s ICER was five-times lower in users who engaged for 52 weeks versus those who engaged less often. Notably, in this study, ‘Limited’ and ‘Occasional’ app users incurred costs without benefit. Deterministic sensitivity analyses revealed that estimated cost-effectiveness was most influenced by the relative risk of diabetes. The probabilistic sensitivity analysis revealed that varying parameter estimates across a wide range of uncertainty mostly resulted in ICERs below the $50,000 WTP threshold, and a small number of iterations resulted in ICERs above this threshold. Taken together, our results suggest that an mHealth app with incentives may be most cost-effective for working aged (20 to 64 yrs) and older (65 + yrs) females. As well, continued efforts to maximize app engagement (e.g., with regular behavioural science-informed feature upgrades)  and minimize reward magnitudes (e.g., by weaning users off daily incentives after 3–4 months)  may yield greater cost-effectiveness.
Beyond the dearth of cost-effectiveness evaluations of commercial physical activity apps, a few related studies help put our results in context. Cost-effectiveness evaluations of physical activity interventions in general have mostly determined that pedometer-based interventions are most cost-effective in Australia, Belgium and the Netherlands (with ICERs ranging from €11,100/QALY to dominant) [73,74,75]. For example, one pedometer-based intervention reported an estimated cost-savings of €500 per person and 0.11–0.16 QALYs gained (more favorable than reported for Carrot Rewards) . Similarly, workplace physical activity incentive programmes have proved to be cost-effective in the UK (with ICERs ranging from not cost-effective to £2900/QALY) [76, 77]. For example, a cost-effectiveness study of a workplace physical activity incentive programme reported estimated incremental costs of £4100 and 1.2 QALYs gained . The considerable disparity in study designs generally makes it difficult to draw direct comparisons to the present study. For instance, the current mHealth intervention was delivered on a population-scale (vs. similar studies that base predictions on pilot data),  used an objectively-measured physical activity outcome (vs. similar studies that used change estimates from separate meta-analyses, or self-report), [73,74,75] and had a younger sample (vs. older samples in similar studies which increases cost-effectiveness) [74, 78] making head-to-head comparisons difficult. We should note that while Carrot Rewards had a relatively small impact on overall QALYs, the impact was greater amongst ‘Committed’ and older users. This is somewhat comparable to previous studies who reported QALY increases of 0.16 and 0.11 for males and females, respectively, but who used a longer time horizon and whose risk reductions were not based on objectively-measured step count increases.58
Limitations and future directions
We made multiple conservative decisions in how our model was structured. First, given that death was a possible outcome, our decision to employ a 5-year time horizon may omit longer term consequences. Extending the horizon may lead to more favorable ICERs. Second, our decision to not include co-morbidities may have also led to less favorable ICERs, as co-morbidities would increase medical costs while decreasing quality of life. Therefore, the positive effects of physical activity delaying disease onset would have produced more favorable results. Finally, we did not include all chronic diseases associated with physical inactivity (e.g., mood and anxiety disorders) . The inclusion of these diseases and their physical activity-related risk reductions into the model would have resulted in more favorable ICERs. This is notable as 27.3% of Carrot Rewards users self-reported a physician diagnosed mood and anxiety disorder after the current study period (unpublished). Therefore, since we assumed all users started in a healthy state, we may have underestimated cost-effectiveness. As we were not able to link participants’ diagnoses with their objectively-measured physical activity, future studies should also consider the health and economic outcomes of similar interventions stratified by health status. We also made some assumptions to convert objectively-measured daily step count increases to risk reductions. First, we assumed a direct linear relationship between physical activity and risk reduction when in fact it is curvilinear with greater risk reduction at lower initial doses (e.g., going from 0 to 30 min as opposed to from 120 to 150 min). This suggests that our model may be underestimating risk reduction and cost-effectiveness given the generally low baseline physical activity levels of our cohort (i.e. 43% accumulated less than 5000 steps per day) . Second, while our intervention was 12 months-long we assumed physical activity increases persisted for five years, as others have done with similar or shorter duration interventions [21, 73,74,75]. Third, while conservative, we assumed daily step count increases occurred at the lower end of moderate intensity  which may not have been the case. Future economic evaluations of accelerometer-based interventions should also take differences in physical activity intensity into consideration when estimating effects on risk reduction.
We also took the perspective of a publicly funded healthcare system as rewards were funded by government partners. A societal perspective would have more favorable ICERs. Our model also includes programmatic costs of the financial incentives only. The model does not include company overhead, as company overhead was funded through other (non-government) revenue sources, or development costs as they were viewed as sunk costs—not normally included as they have no impact on the marginal costs and benefits of continuing an intervention .
This is the first cost-effectiveness study of a commercially available physical activity app. We have shown that an incentive-based mHealth app targeting physical activity would be cost-effective in two Canadian provinces over a five-year time horizon. As digital health technologies continue to evolve to address emerging and persistent global health issues, it is incumbent on researchers and policy-makers to demand a broader evidence-base that includes health economic impacts to inform public health policy decisions.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
World Health Organization
Incremental cost-effectiveness ratio
Monthly active user
Randomized controlled trial
Quality-adjusted life years
Consolidated Health Economic Evaluation Reporting Standards
Ischemic heart disease
Colley RC, Garriguet D, Janssen I, Craig CL, Clarke J, Tremblay MS. Physical activity of Canadian adults: accelerometer results from the 2007 to 2009 Canadian health measures survey. Heal reports. 2011;22(1):7.
Althoff T, Hicks JL, King AC, Delp SL, Leskovec J. Large-scale physical activity data reveal worldwide activity inequality. Nature. 2017;547(7663):336–9. https://doi.org/10.1038/nature23018.
Bassett DR Jr, Wyatt HR, Thompson H, Peters JC, Hill JO. Pedometer-measured physical activity and health behaviors in United States adults. Med Sci Sports Exerc. 2010;42(10):1819–25. https://doi.org/10.1249/MSS.0b013e3181dc2e54.
Kohl HW 3rd, Craig CL, Lambert EV, Inoue S, Alkandari JR, Leetongin G, et al. The pandemic of physical inactivity: global action for public health. Lancet. 2012;380(9838):294–305. https://doi.org/10.1016/S0140-6736(12)60898-8.
Krueger H, Turner D, Krueger J, Ready AE. The economic benefits of risk factor reduction in Canada: tobacco smoking, excess weight and physical inactivity. Can J Public Heal. 2014;105(1):e69–78. https://doi.org/10.17269/cjph.105.4084.
Ding D, Lawson KD, Kolbe-Alexander TL, Finkelstein EA, Katzmarzyk PT, Van Mechelen W, et al. The economic burden of physical inactivity: a global analysis of major non-communicable diseases. Lancet. 2016;388(10051):1311–24. https://doi.org/10.1016/S0140-6736(16)30383-X.
World Health Organization (WHO). Global action plan on physical activity 2018–2030: more active people for a healthier world. Geneva: World Health Organization; 2018.
National Healthy Living Platform: “Carrot Rewards” targets lifestyle improvements - Canada.ca [Internet]. [cited 2021 Oct 4]. Available from: https://www.canada.ca/en/news/archive/2015/07/national-healthy-living-platform-carrot-rewards-targets-lifestyle-improvements.html
Smart Applications - Ministry of Health and Prevention - UAE [Internet]. [cited 2021 Oct 4]. Available from: https://www.mohap.gov.ae/en/services/Pages/DownloadApps.aspx
South Africa Digital Health Dashboard | HealthEnabled [Internet]. [cited 2021 Oct 4]. Available from: http://healthenabled.org/wordpress/south-africa-digital-health-dashboard/
Willis Towers Watson. Willis Towers Watson’s 23rd Annual Best Practices in Health Care Employer Survey. 2018.
Mercer. Europe Corporate Wellness Trends: How Benefits Are Shaping the Future of Work. 2019.
Feter N, Dos Santos TS, Caputo EL, da Silva MC. What is the role of smartphones on physical activity promotion? A systematic review and meta-analysis. Int J Public Health. 2019;64(5):679–90. https://doi.org/10.1007/s00038-019-01210-7.
Petersen JM, Prichard I, Kemps E. A comparison of physical activity mobile apps with and without existing web-based social networking platforms: systematic review. J Med Internet Res. 2019;21(8):e12687. https://doi.org/10.2196/12687.
Romeo A, Edney S, Plotnikoff R, Curtis R, Ryan J, Sanders I, et al. Can smartphone apps increase physical activity? Systematic review and meta-analysis. J Med Internet Res. 2019;21(3):e12053. https://doi.org/10.2196/12053.
Bondaronek P, Alkhaldi G, Slee A, Hamilton FL, Murray E. Quality of publicly available physical activity apps: review and content analysis. JMIR mHealth uHealth. 2018;6(3):e9069. https://doi.org/10.2196/mhealth.9069.
Schoeppe S, Alley S, Van Lippevelde W, Bray NA, Williams SL, Duncan MJ, et al. Efficacy of interventions that use apps to improve diet, physical activity and sedentary behaviour: a systematic review. Int J Behav Nutr Phys Act. 2016;13(1):1–26. https://doi.org/10.1186/s12966-016-0454-y.
Rahimi K. Digital health and the elusive quest for cost savings. Lancet Digit Heal. 2019;1(3):e108–9. https://doi.org/10.1016/S2589-7500(19)30056-1.
Iribarren SJ, Cato K, Falzon L, Stone PW. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PLoS One. 2017;12(2):e0170581. https://doi.org/10.1371/journal.pone.0170581.
National Institute for Health and Care Excellence. Evidence Standards Framework for Digital Health Technologies. 2019.
Roux L, Pratt M, Tengs TO, Yore MM, Yanagawa TL, Van Den Bos J, et al. Cost effectiveness of community-based physical activity interventions. Am J Prev Med. 2008;35(6):578–88. https://doi.org/10.1016/j.amepre.2008.06.040.
Shah GH, Bogaert K, Yin J, Gould E, Leider JP. Gaps in public health workers’ awareness of emerging public health trends. J Public Heal Manag Pract. 2020;26(5):401–3. https://doi.org/10.1097/PHH.0000000000001049.
Peloton app installs rise 245%, but does it mean anything? - Business of Apps [Internet]. [cited 2021 Oct 4]. Available from: https://www.businessofapps.com/insights/peloton-app-installs-rise-245-but-does-it-mean-anything/
World Health Organization (WHO). mHealth: New horizons for health through mobile technologies. Vol. 3. Global Observatory for eHealth series; 2011.
Research2Guidance. mHealth Economics 2017/2018: Current Status and Future Trends in Mobile Health 2017.
Laine J, Kuvaja-Köllner V, Pietilä E, Koivuneva M, Valtonen H, Kankaanpää E. Cost-effectiveness of population-level physical activity interventions: a systematic review. Am J Health Promot. 2014;29(2):71–80. https://doi.org/10.4278/ajhp.131210-LIT-622.
Abu-Omar K, Rütten A, Burlacu I, Schätzlein V, Messing S, Suhrcke M. The cost-effectiveness of physical activity interventions: a systematic review of reviews. Prev Med reports. 2017;8:72–8. https://doi.org/10.1016/j.pmedr.2017.08.006.
Mizdrak A, Telfer K, Direito A, Cobiac LJ, Blakely T, Cleghorn CL, et al. Health gain, cost impacts, and cost-effectiveness of a mass media campaign to promote smartphone apps for physical activity: modeling study. JMIR mHealth uHealth. 2020;8(6):e18014. https://doi.org/10.2196/18014.
Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot. 1997;12(1):38–48. https://doi.org/10.4278/0890-1171-12.1.38.
Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, et al. Consolidated health economic evaluation reporting standards (CHEERS) statement. Int J Technol Assess Health Care. 2013;29(2):117–22. https://doi.org/10.1017/S0266462313000160.
Mitchell M, Lau E, White L, Faulkner G. Commercial app use linked with sustained physical activity in two Canadian provinces: a 12-month quasi-experimental study. Int J Behav Nutr Phys Act. 2020;17(1):1–9. https://doi.org/10.1186/s12966-020-00926-7.
Rondina R II, Pearson EK, Prapavessis H, White L, Richard S, Mitchell MS. Bright spots, physical activity investments that (almost) worked: carrot rewards app, driving engagement with pennies a day. Br J Sports Med. 2020;54(15):927–9. https://doi.org/10.1136/bjsports-2019-100969.
Mitchell MS, Orstad SL, Biswas A, Oh PI, Jay M, Pakosh MT, et al. Financial incentives for physical activity in adults: systematic review and meta-analysis. Br J Sports Med. 2020;54(21):1259–68. https://doi.org/10.1136/bjsports-2019-100633.
Toronto tech company plans to relaunch Carrot Rewards wellness app - The Globe and Mail [Internet]. [cited 2021 Oct 4]. Available from: https://www.theglobeandmail.com/business/technology/article-toronto-tech-company-plans-to-relaunch-carrot-rewards-wellness-app/
Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 4th Edition. Ottawa; 2017.
Janssen I. Health care costs of physical inactivity in Canadian adults. Appl Physiol Nutr Metab. 2012;37(4):803–6. https://doi.org/10.1139/h2012-061.
Warburton DER, Charlesworth S, Ivey A, Nettlefold L, Bredin SSD. A systematic review of the evidence for Canada’s physical activity guidelines for adults. Int J Behav Nutr Phys Act. 2010;7(1):1–220. https://doi.org/10.1186/1479-5868-7-39.
Riebe D, Ehrman JK, Liguori G, Magal M. Medicine AC of S. ACSM’s guidelines for exercise testing and prescription. Wolters Kluwer; 2018.
Statistics Canada: Canada’s national statistical agency [Internet]. [cited 2018 Oct 4]. Available from: https://www.statcan.gc.ca/eng/start
Canadian Chronic Disease Surveillance System (CCDSS) [Internet]. [cited 2018 Oct 4]. Available from: https://health-infobase.canada.ca/ccdss/data-tool/
Wu S, Cohen D, Shi Y, Pearson M, Sturm R. Economic analysis of physical activity interventions. Am J Prev Med. 2011;40(2):149–58. https://doi.org/10.1016/j.amepre.2010.10.029.
Chaudhry UAR, Wahlich C, Fortescue R, Cook DG, Knightly R, Harris T. The effects of step-count monitoring interventions on physical activity: systematic review and meta-analysis of community-based randomised controlled trials in adults. Int J Behav Nutr Phys Act. 2020;17(1):1–16. https://doi.org/10.1186/s12966-020-01020-8.
Sundquist K, Qvist J, Johansson SE, Sundquist J. The long-term effect of physical activity on incidence of coronary heart disease: a 12-year follow-up study. Prev Med (Baltim). 2005;41(1):219–25. https://doi.org/10.1016/j.ypmed.2004.09.043.
Mai PL, Sullivan-Halley J, Ursin G, Stram DO, Deapen D, Villaluna D, et al. Physical activity and colon cancer risk among women in the California teachers study. Cancer Epidemiol Biomark Prev. 2007;16(3):517–25. https://doi.org/10.1158/1055-9965.EPI-06-0747.
Friedenreich C, Norat T, Steindorf K, Boutron-Ruault MC, Pischon T, Mazuir M, et al. Physical activity and risk of colon and rectal cancers: the European prospective investigation into Cancer and nutrition. Cancer Epidemiol Biomark Prev. 2006;15(12):2398–407. https://doi.org/10.1158/1055-9965.EPI-06-0595.
Myint PK, Luben RN, Wareham NJ, Welch AA, Bingham SA, Day NE, et al. Combined work and leisure physical activity and risk of stroke in men and women in the European prospective investigation into cancer-Norfolk prospective population study. Neuroepidemiology. 2006;27(3):122–9. https://doi.org/10.1159/000095551.
Calling S, bo H, Engström G, Berglund G, Janzon L. Effects of body fatness and physical activity on cardiovascular risk: risk prediction using the bioelectrical impedance method. Scand J Public Health. 2006;34(6):568–75. https://doi.org/10.1080/14034940600595621.
Armstrong MEG, Green J, Reeves GK, Beral V, Cairns BJ. Frequent physical activity may not reduce vascular disease risk as much as moderate activity: large prospective study of women in the United Kingdom. Circulation. 2015;131(8):721–9. https://doi.org/10.1161/CIRCULATIONAHA.114.010296.
Folsom AR, Kushi LH, Hong CP. Physical activity and incident diabetes mellitus in postmenopausal women. Am J Public Health. 2000;90(1):134–8. https://doi.org/10.2105/ajph.90.1.134.
Howard RA, Freedman DM, Park Y, Hollenbeck A, Schatzkin A, Leitzmann MF. Physical activity, sedentary behavior, and the risk of colon and rectal cancer in the NIH-AARP diet and health study. Cancer Causes Control. 2008;19(9):939–53. https://doi.org/10.1007/s10552-008-9159-0.
Hu FB, Leitzmann MF, Stampfer MJ, Colditz GA, Willett WC, Rimm EB. Physical activity and television watching in relation to risk for type 2 diabetes mellitus in men. Arch Intern Med. 2001;161(12):1542–8. https://doi.org/10.1001/archinte.161.12.1542.
Rockhill B, Willett WC, Hunter DJ, Manson JE, Hankison SE, Colditz GA. A prospective study of occupational physical activity and breast cancer risk. Cancer Causes Control. 2015;26(12):1779–89. https://doi.org/10.1007/s10552-015-0671-8.
McTiernan A, Kooperberg C, White E, Vogel VG. Recreational physical activity and the risk of breast cancer in postmenopausal women: the women’s health initiative cohort study. Breast Dis. 2004;15(1):25–6. https://doi.org/10.1097/01.OGX.0000115859.71446.89.
Okada K, Hayashi T, Tsumura K, Suematsu C, Endo G, Fujii S. Leisure-time physical activity at weekends and the risk of type 2 diabetes mellitus in Japanese men: the Osaka health survey. Diabet Med. 2000;17(1):53–8. https://doi.org/10.1046/j.1464-5491.2000.00229.x.
Tanasescu M, Leitzmann MF, Rimm EB, Willett WC, Stampfer MJ, Hu FB. Exercise type and intensity in relation to coronary heart disease in men. J Am Med Assoc. 2002;288(16):1994–2000. https://doi.org/10.1001/jama.288.16.1994.
Economic Burden of Illness in Canada - Canada.ca [Internet]. [cited 2018 Oct 4]. Available from: https://www.canada.ca/en/public-health/services/chronic-diseases/chronic-disease-knowledge-development-exchange/economic-burden-illness-canada.html
Golicki D, Niewada M, Buczek J, Karlińska A, Kobayashi A, Janssen MF, et al. Validity of EQ-5D-5L in stroke. Qual Life Res. 2015;24(4):845–50. https://doi.org/10.1007/s11136-014-0834-1.
Färkkilä N, Sintonen H, Saarto T, Järvinen H, Hänninen J, Taari K, et al. Health-related quality of life in colorectal cancer. Color Dis. 2013;15(5):e215–22. https://doi.org/10.1111/codi.12143.
Yousefi M, Najafi S, Ghaffari S, Mahboub-Ahari A, Ghaderi H. Comparison of SF-6D and EQ-5D scores in patients with breast cancer. Iran Red Crescent Med J. 2016;18(5):e23556.
Bansback N, Tsuchiya A, Brazier J, Anis A. Canadian valuation of EQ-5D health states: Preliminary value set and considerations for future valuation studies. PLoS One. 2012;7(2):e31115.
Tsiplova K, Pullenayegum E, Cooke T, Xie F. EQ-5D-derived health utilities and minimally important differences for chronic health conditions: 2011 Commonwealth Fund survey of sicker adults in Canada. Qual Life Res. 2016;25(12):3009–16. https://doi.org/10.1007/s11136-016-1336-0.
Briggs A, Sculpher M. Claxton K. Oup Oxford: Decision modelling for health economic evaluation; 2006.
GDP per capita (current US$) - Canada | Data [Internet]. [cited 2021 Oct 4]. Available from: https://data.worldbank.org/indicator/NY.GDP.PCAP.CD?locations=CA
Brower J, LaBarge MC, White L, Mitchell MS. Examining responsiveness to an incentive-based Mobile health app: longitudinal observational study. J Med Internet Res. 2020;22(8):e16797. https://doi.org/10.2196/16797.
Adams J, Giles EL, McColl E, Sniehotta FF. Carrots, sticks and health behaviours: a framework for documenting the complexity of financial incentive interventions to change health behaviours. Health Psychol Rev. 2014;8(3):286–95. https://doi.org/10.1080/17437199.2013.848410.
Mitchell MS, Goodman JM, Alter DA, Oh PI, Faulkner GEJ. Development of the health incentive program questionnaire (HIP-Q) in a cardiac rehabilitation population. Transl Behav Med. 2015;5(4):443–59. https://doi.org/10.1007/s13142-015-0330-3.
Camerer CF, Loewenstein G. Behavioral economics: past, present, future. In: Advances in Behavioral Economics, Princeton, Princeton University Press Chang, H (2000)‘A Liberal Theory of Social Welfare: Fairness, Utility, and the Pareto Principle’, Yale Law Review. Citeseer; 2003.
Wertenbroch K, Soman D, Chattopadhyay A. On the perceived value of money: the reference dependence of currency numerosity effects. J Consum Res. 2007;34(1):1–10. https://doi.org/10.1086/513041.
Carrera M, Royer H, Stehr M, Sydnor J. Can financial incentives help people trying to establish new habits? Experimental evidence with new gym members. J Health Econ. 2018;58:202–14. https://doi.org/10.1016/j.jhealeco.2018.02.010.
Condliffe S, Işgın E, Fitzgerald B. Get thee to the gym! A field experiment on improving exercise habits. J Behav Exp Econ. 2017;70:23–32. https://doi.org/10.1016/j.socec.2017.07.007.
Rohde KIM, Verbeke W. We like to see you in the gym—a field experiment on financial incentives for short and long term gym attendance. J Econ Behav Organ. 2017;134:388–407. https://doi.org/10.1016/j.jebo.2016.12.012.
Leahey TM, Subak LL, Fava J, Schembri M, Thomas G, Xu X, et al. Benefits of adding small financial incentives or optional group meetings to a web-based statewide obesity initiative. Obesity. 2015;23(1):70–6. https://doi.org/10.1002/oby.20937.
Cobiac LJ, Vos T, Barendregt JJ. Cost-effectiveness of interventions to promote physical activity: a modelling study. PLoS Med. 2009;6(7):e1000110. https://doi.org/10.1371/journal.pmed.1000110.
De Smedt D, De Cocker K, Annemans L, De Bourdeaudhuij I, Cardon G. A cost-effectiveness study of the community-based intervention ‘10 000 Steps Ghent.’. Public Health Nutr. 2012;15(3):442–51. https://doi.org/10.1017/S1368980011001716.
Over EAB, Wendel-Vos GCW, van den Berg M, Hamberg-van Reenen HH, Tariq L, Hoogenveen RT, et al. Cost-effectiveness of counseling and pedometer use to increase physical activity in the Netherlands: a modeling study. Cost Eff Resour Alloc. 2012;10(1):1–7. https://doi.org/10.1186/1478-7547-10-13.
Hunter RF, Murray JM, Gough A, Tang J, Patterson CC, French DP, et al. Effectiveness and cost-effectiveness of a loyalty scheme for physical activity behaviour change maintenance: results from a cluster randomised controlled trial. Int J Behav Nutr Phys Act. 2018;15(1):1–13. https://doi.org/10.1186/s12966-018-0758-1.
Dallat MAT, Hunter RF, Tully MA, Cairns KJ, Kee F. A lesson in business: cost-effectiveness analysis of a novel financial incentive intervention for increasing physical activity in the workplace. BMC Public Health. 2013;13(1):1–9. https://doi.org/10.1186/1471-2458-13-953.
Frew EJ, Bhatti M, Win K, Sitch A, Lyon A, Pallan M, et al. Cost-effectiveness of a community-based physical activity programme for adults (be active) in the UK: an economic analysis within a natural experiment. Br J Sports Med. 2014;48(3):207–12. https://doi.org/10.1136/bjsports-2012-091202.
Schuch FB, Vancampfort D, Firth J, Rosenbaum S, Ward PB, Silva ES, et al. Physical activity and incident depression: a meta-analysis of prospective cohort studies. Am J Psychiatry. 2018;175(7):631–48. https://doi.org/10.1176/appi.ajp.2018.17111194.
Cellini SR, Kee JE. Cost-effectiveness and cost-benefit analysis. Handb Pract Progr Eval. 2010;3:493-560.
We acknowledge Guy Faulkner and Erica Lau for providing the point estimates that were previously published in our other studies and used as inputs for the model of the current paper.
The Public Health Agency of Canada (PHAC) Multi-Sectoral Partnership Approach to Healthy Living and Chronic Disease Prevention grant, as well as the Baycrest Hospital’s Soupcoff Family Fellowship, the Mitacs Elevate program, the Sandra Rotman Centre for Health Sector Strategy Fellowship, and the Social Sciences and Humanities Research Council Partnership Grant. The funding bodies did not contribute to any aspect of the study.
Ethics approval and consent to participate
Consent for publication
MM received consulting fees from Carrot Insights Inc., makers of the Carrot Rewards app, from 2015 to 2018 as well as travel re-imbursement in January and March 2019. MM had stock options in the company as well but these are now void since Carrot Insights Inc. went out of business in June 2019. RR was employed as a research intern by Carrot Insights Inc. from March 2018 to July 2018, and completed a Mitacs postdoctoral fellowship at the company from February 2019 to June 2019. MH and SS declare that they have no conflicts of interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Detailed description of methods, cohort, and data sources.
Cohort composition by age, gender, geography, and engagement level.
Illustration of Markov model design: in this state-transition diagram all of the seven health states are inserted. Each arrow is linked with a certain transition probability. Circles represent possible health states. The following states can be distinguished: healthy, diabetes, colorectal cancer, breast cancer, (ischemic) heart disease, stroke, and death.
Age-, gender-, and geography-dependent transition probabilities by chronic disease.
Age-, gender-, and disease-dependent mortality rates by health state.
Relative risk by engagement level, age group, and gender.
Age-, gender-, and geography dependent average annual medical costs by chronic disease.
Age-, gender-, and geography-dependent average annual intervention costs by engagement level.
Age-, gender-, and geography dependent average annual medical costs by chronic disease.
Lower and upper bounds for each parameter in the deterministic sensitivity analysis.
About this article
Cite this article
Rondina, R., Hong, M., Sarma, S. et al. Is it worth it? Cost-effectiveness analysis of a commercial physical activity app. BMC Public Health 21, 1950 (2021). https://doi.org/10.1186/s12889-021-11988-y
- Public health
- Physical activity
- Behavioural economics