Skip to main content
Download PDF

Assessing the epidemiological impact on cervical cancer of switching from 4-valent to 9-valent HPV vaccine within a gender-neutral vaccination programme in Switzerland

BMC Public Health volume 20, Article number: 671 (2020) Cite this article

Abstract

Background

An infection with high-risk human papillomavirus (HPV) is the obligatory aetiological factor for the development of cervical cancer. In Switzerland, the prevention strategy for cervical cancer is based on primary prevention via HPV vaccination and secondary prevention with an opportunistic screening programme for precancerous lesions. Vaccination is recommended to 11–26 years old male and female persons. The objective of the study was to assess the epidemiological impact on cervical cancer of switching from the currently implemented programme with the 4-valent vaccine to the 9-valent vaccine, in an 11–26 years old gender-neutral vaccination programme in Switzerland.

Methods

A previously validated dynamic transmission model of HPV infections was adapted and calibrated to the Swiss setting assuming an 80% coverage rate in HPV-vaccination and lifelong vaccine type-specific protection. A gender-neutral vaccination programme (males and females) for 11–26 years old with a 9-valent HPV vaccine was compared with the current 11–26 years old gender-neutral 4-valent vaccination programme. Sensitivity analyses were conducted in order to test the impact of lower vaccination coverage rates and a shorter duration of protection on the model outcomes.

Results

In Switzerland, a 9-valent gender-neutral vaccination programme would result in an additional prevention of 2979 cervical cancer cases, 13,862 CIN3 and 15,000 CIN2 cases, compared with the 4-valent gender-neutral vaccination programme over 100 years. These additional disease cases avoided would correspond to a 24, 36 and 48% cumulative incidence decrease in cervical cancer, CIN3 and CIN2 cases, respectively. It would also prevent additional 741 cervical cancer-related deaths over 100 years. A substantial additional reduction in cervical cancer and precancerous lesions burden is still observed when varying the vaccination coverage rate from 30 to 60% or reducing the duration of protection from lifelong to 20 years.

Conclusions

The switch to the 9-valent vaccine in Switzerland to prevent cervical diseases showed an important contribution in terms of public health impact compared with the 4-valent vaccine in an 11–26 years old gender-neutral population, even with very conservative assumptions such as low coverage rates or low duration of protection and limiting analysis to only cervical disease.

Peer Review reports

Background

Human papillomavirus (HPV) is the most common sexually transmitted infection. Although most HPV infections are transient and clear up without intervention within a few months after acquisition, persistent HPV infections may cause pre-cancerous lesions and ultimately cancer. More than 200 HPV genotypes have been identified. At least 12 off these are classified as oncogenic High-Risk types (types 16/18/31/33/35/39/45/51/52/56/58/59), which are causal agents of HPV-related cancers including cervical, vaginal, vulvar, penile, anal and head and neck cancer [1, 2]. On the other hand, low-risk HPV types such as HPV 6 and HPV 11 cause 90% of genital warts as well as the rare but debilitating recurrent respiratory papillomatosis (RRP) [3, 4]. In Europe, HPV is associated with almost 100% of cervical cancer cases and causes around 87% of anal, 70% of vaginal, 29% of penile, 16% of vulvar and 7% of head and neck cancers in females and males [4].

The annual number of new disease cases related to HPV in Europe were estimated to be close to 35,000 cases of cervical cancer, 1500 cases of vulvar cancer, 1500 cases of vaginal cancer, 6500 cases of anal cancer, 1200 cases of penile cancer and 7000 cases of head and neck cancer [4].

In Switzerland, around 5000 new cases of cervical precancerous lesions (CIN2 and CIN3) and 250 new cases of cervical cancer are being diagnosed and result in an estimated 75 deaths each year. Cervical cancer is the fifth most frequent type of cancer among women aged 20 to 49 years [5, 6].

HPV 16/18 are the predominant HPV types in cervical diseases: 23–25% of CIN1 lesions, 38.4–39% of CIN2 lesions, 58% of CIN3 lesions and 73% of cervical cancer cases are caused by these two HPV types. The HPV 31/33/45/52/58 attributable fraction among HPV positive cases is estimated to be around 23–26% in CIN1 lesions, 32–35.9% in CIN2 lesions, 27–32% in CIN3 lesions and 16.2% in cervical cancer cases [7]. These international data on HPV attributable fraction have been confirmed in the Swiss setting. A recent study aiming to identify the HPV types present in the cervix tissue was conducted among 768 Swiss women diagnosed with CIN3+ between 2014 and 2015. The analysis showed that 61.8% of CIN3+ cases were associated with HPV 16/18, with an attributable fraction of HPV 31/33/45/52/58 of 27.7% (89.5% of cases were associated with an HPV genotype included in the 9-valent vaccine) [8, 9].

In Switzerland, the prevention strategy for cervical cancer is based on gender-neutral HPV vaccination (primary prevention) and screening (secondary prevention).

Cervical cancer screening in Switzerland is not provided through a national organized programme but on an opportunistic basis, meaning that cervical cancer screening is left to the initiative of women and doctors. However, cervical cancer screening based on cytology is recommended every 3 years for women aged 21 to 30 years and screening based on cytology or on primary HPV testing is recommended every 3 years for women aged 31–70 years [10]. Cervical cancer screening based on cytology for women aged 21 to 70 is reimbursed by Swiss health insurances.

In addition to screening, preventive gender-neutral HPV vaccination exists. Currently, there are three HPV vaccines approved in Switzerland. Two vaccines, targeting HPV 6/11/16/18 and HPV 16/18 were first licensed for clinical use in 2006 and 2010, respectively (Gardasil®, MSD, 4-valent HPV vaccine and Cervarix®, GSK SPC, 2-valent HPV vaccine) [11, 12]. Although both 4-valent and 2-valent HPV vaccines are recommended and approved in Switzerland, the 4-valent vaccine leads the HPV vaccines market with almost 95% of HPV vaccinations done with this vaccine (based on IMS data of MAT June 2017) [13]. This is mainly due to the fact that the 4-valent vaccine is the only HPV vaccine approved for the prevention of HPV-related diseases in boys and young men in Switzerland [11, 12]. A third vaccine (Gardasil 9®, MSD), was approved by the Swiss regulatory body Swissmedic in July 2016 for protection against nine HPV types (6, 11, 16, 18, 31, 33, 45, 52 and 58) and expected to provide increased coverage against the majority of high-risk HPV types with carcinogenic properties [14].

HPV vaccination was first recommended in Switzerland to girls and young women in 2007, and was then extended to boys and young men in 2015 by the Swiss Federal Office of Public Health (BAG) and the Federal Vaccination Commission (EKIF). The BAG and EKIF stated the following reasons for switching to a gender-neutral vaccination programme: a significant burden of HPV-associated diseases including genital warts occurring in men, protection of specific risk groups (men having sex with men - MSM), good safety and efficacy profile of HPV vaccination in males, and improved herd protection for both male and female. With the introduction of gender-neutral vaccination, men were no longer excluded from the benefits of the HPV vaccination [15]. It was also argued that gender-neutral HPV vaccination allows both sexes to bear responsibility for issues around sexual and reproductive health [15]. Recently other countries in Europe have also expanded their programme to a gender-neutral vaccination (e.g. Germany, Ireland and UK) [16,17,18,19]. The Swiss HPV vaccination programme is based on a two-dose regimen for girls and boys aged 11–14 years and a three-dose regimen for persons aged 15 to 26 years. Vaccination uptake and coverage rate in Switzerland was estimated around 69 and 54% respectively among 18–20 year old and close to 42 and 34% among 21–24 year old [20]. Due to the fact that the HPV vaccination programme is implemented on the cantonal level, a substantial variability in the HPV vaccination coverage rate in girls aged 16 is observed among the 26 Cantons, ranging from 30% in the Canton of Obwalden to 79% in the Canton of Valais [21].

The epidemiological impact of HPV vaccination on incidence and prevalence of HPV infections and HPV-related diseases has been extensively studied following the change of vaccination programme from a 4-valent to a 9-valent vaccine in many countries. Several European mathematical models have estimated the potential impact of switching from a 4-valent to a 9-valent HPV vaccination on the burden of HPV-related diseases or infections [22,23,24]. These studies, conducted in Austria, Germany and Italy, provided similar conclusions on the positive impact of the switch to a gender-neutral 9-valent vaccination on HPV-related diseases such as cervical cancer and pre-cancerous lesions [22,23,24].

In Switzerland, no such analysis on the epidemiological benefits of a 9-valent vaccination compared with the current vaccination programme has been done so far. Thus, the objective of this analysis was to estimate the epidemiological impact of switching to a 9-valent HPV vaccination programme for 11–26 years old boys and girls from the currently implemented programme with the 4-valent vaccination in Switzerland. This analysis considered the currently available 4-valent vaccine as the relevant comparator since almost 95% of current HPV vaccination is performed with this vaccine in Switzerland [13].

Methods

Mathematical model

A previously published and validated transmission model, simulating the natural history of genotypes 6/11/16/18 HPV-infections, has been extended to account for infections and diseases attributable to HPV genotypes 31/33/45/52/58. A detailed description of the model can be found in Boiron et al. [21] and Elbasha et al. [24,25,26] In summary, the dynamic model structure accounts for herd protection effects and herd immunity.).

An epidemiological module simulates HPV types 6, 11, 16 and 18 separately, whereas the five additional types (31, 33, 45, 52 and 58) are combined into a single set of compartments. The model accounts for the transmission dynamics of those nine HPV types and simulates the occurrence of genital warts; RRP; pre-cancerous lesions (CIN1–3); cervical, vulvar, vaginal, penile, anal, and head and neck cancers related to these HPV types.

For the Swiss adaptation of the model, only the following HPV-related diseases were included in the analysis: cervical cancer, CIN1, 2 and 3.

Despite that HPV causes vaginal, vulvar, anal, head and neck, and penile cancers, as well as RRP and genital warts, these diseases were not included in the analysis and will not be presented in the following sections for three main reasons. (i) The estimated incidence of anal cancers attributed to the HPV types 31, 33, 45, 52, 58 is relatively low. The overall incidence of HPV-associated anal cancers and HPV-associated head and neck cancers in Switzerland is estimated to be 155 cases and 73 cases, respectively [4, 15]. 2.9% of HPV-associated anal cancers and 3.7% of HPV-associated oropharyngeal cancers can be attributed to the 5 HPV additional types in the 9-valent vaccine [4]. This means that less than 4 additional anal cancer cases and less than 8 additional oropharyngeal cancer cases could be prevented every year with the 9-valent vaccine compared with the 4-valent vaccine. (ii) While there is no data available on the incidence of RRP, vaginal, vulvar and penile cancer in Switzerland, the estimated number of cases attributed to the HPV types 31, 33, 45, 52, 58 is low (less than 7 cases per year). (iii) 90% of genital warts are caused by HPV types 6 and 11. As both the 4-valent and 9-valent vaccine cover these types, no difference between prevented cases is expected.

Model inputs

The model inputs, including data on demographics, sexual behaviour, screening, natural history of the disease, treatment patterns, cancer mortality and vaccination coverage were adapted to the Swiss setting.

Demographics

Population data was issued from the Federal Office of Statistics (BFS). The total population in Switzerland at the end of 2015 was estimated to be 8,327,126 people [25].

Sexual behaviour

Sexual behavioural data in Switzerland is scarce and therefore the results from the UK-NATSAL III (the third National Survey of Sexual Attitudes and Lifestyles) survey were used [26]. According to expert opinion (AK) this data is applicable to Switzerland. The extent of sexual mixing (value between 0 and 1 with 0 representing no mixing and 1 representing maximum mixing) among members of different age cohorts and sexual activity groups was extracted from the technical model report accompanying the publication by Elbasha et al. “Impact of Vaccinating Boys and Men against HPV in the United States” and adjusted during the calibration process [27].

Screening

The percentage of females receiving gynaecological pre-cancer screening tests at least once every 3 years (75%) was derived from a study by Boiron et al. [22] and the proportion of women who received a follow-up screening test after an abnormal PAP result was set to 95% [28]. Age-specific coverage in the past year were extracted from a study by Burton-Jeangros et al. [29] and validated by expert opinion. In terms of diagnostic performance, sensitivity and specificity values of the screening tests were derived from Elbasha et al. [30].

Natural history of the disease

The progression from infection to disease follows a similar history structure as the initial US model [30]. As transmission rates are not directly observable, the US values were used and adjusted through calibration techniques to obtain the set of parameters that best fit the Swiss data on cervical cancer and associated mortality.

Disease and treatment patterns

The female population receiving hysterectomy over the course of 1 year was estimated from inpatient operation statistics for 2015 published by the Federal Office of Statistics/ Bundesamt für Statistik (BFS). The age-specific incidence of hysterectomies was calculated as the total number of hysterectomies by age divided by the sum of women by age group in Switzerland as reported by the BFS [25].

Mortality

The annual probability of death for each HPV-related cancer, stratified by age and stage, were derived from the Austrian survival age-specific data from the EUROCARE-5 database [31, 32].

Vaccination strategy

Annual HPV 4-valent vaccination coverage data from Switzerland in 2016 were used to inform vaccination coverage rates in the model [21]. It was assumed that 68% of girls aged 11–14 years were vaccinated in the first year. This percentage was based on the 56% of 16-year-old girls having received two doses of the 4-valent HPV vaccine in 2016 and assumes that the vaccination coverage rate (VCR) at launch of the 9-valent vaccine would be higher than the observed rate for the 4-valent vaccine in 2016. For boys aged 11–14 years, the VCR was assumed to be around 20% in the first year of the 9-valent vaccine introduction. A catch-up cohort of girls and boys aged 15–26 who had not been previously vaccinated or had not undergone a complete vaccine schedule was also considered assuming a VCR at launch of the 9-valent vaccine of 45% in women and 10% in men. VCR for both females and males aged 11–26 years was assumed to increase up to 80% in 10 years’ time.

As vaccines are assumed to have a similar clinical effect in all countries, vaccine properties parameters were kept the same as in the core model and were based on clinical trial data (Table 1) [33,34,35,36,37,38]. The duration of protection against HPV genotypes included in the 9-valent vaccine was assumed to be lifelong in the base case scenario. The efficacy for one dose was assumed to be 0%, efficacy for two doses was assumed to be 100% and herd immunity was considered. Compliance for second dose was assumed to be 95%. For people aged 15–26 an additional third dose was considered in the model as recommended by the Swiss vaccination schedule. Efficacy and compliance for this third dose was assumed at 100%.

Table 1 Summary table on vaccine efficacy assumptions for HPV-related cervical cancer
Full size table

Model calibration and validation

The calibration process aims to tune epidemiological inputs in order to obtain results closer to observed data. The targets of the calibration are the incidence and mortality rates of HPV-related diseases. Swiss HPV-related diseases overall incidence and mortality data were collected from the National Institute for Cancer Epidemiology and Registration (NICER) [39, 40] and the Federal Office of Public Health (BAG) [41] websites. Since the model studies HPV-induced cancers only, target epidemiological values were calculated by multiplying the incidence or the mortality with the percentage of the disease that can be attributed to HPV infection. The proportions of diseases attributable to HPV infection were collected from two publications by Hartwig et al. [42, 43].

The calibration process involved many rounds of iterations to move model outcomes closer to the targets. The following model outcomes were compared against the calibration target in each iteration:

  • Cervical cancer incidence

  • Mortality rate of cervical cancer

Several model variables (including behavioural parameters, natural history of disease, transmission rates, and all-cause mortality) were first adjusted by changing transmission rates. Then the specific parameters (e.g. rate of mortality and rate of seeking treatment for most cancers) were used to fine-tune each disease area.

Epidemiological impact analysis

Two strategies were compared in this study:

  • 9-valent vaccination for girls and boys aged 11–26 years in addition to the current cervical screening programme

  • 4-valent vaccination for girls and boys aged 11–26 years in addition to the current cervical screening programme

The epidemiological impact of HPV vaccination strategies was calculated by estimating the total number of disease events prevented related to the HPV types 16/18/31/33/45/52/58; the reduction in HPV 16/18/31/33/45/52/58 infection prevalence; the incidence and mortality reduction of cervical cancer and the incidence reduction of CIN1–3. The cumulative percentage reduction of 9-valent vaccination versus 4-valent vaccination was calculated by estimating the incidence of each HPV-related disease for each year of the simulation for each scenario, and then by calculating the difference of actual number of cases of each HPV-related disease between the two scenarios on the population level and over the entire time period (usually 25, 50 and 100 years). Results were reported at 25 years, 50 years and 100 years for the different strategies tested.

Sensitivity analyses were performed deterministically, modifying the value of one base case parameter at a time. The two key parameters tested in this analysis are the maximum vaccination coverage rate (60% in males and females, 45% in males and females, 45% in females and 30% in males, 30% in males and females) and the duration of protection (20 years).

Results

Calibration

The model accurately replicated the published cervical cancer incidence and mortality rates (less than 1% difference). Indeed, the model reported the same values than observed cervical cancer incidence and mortality in females for the 4-valent vaccine (4.25/100,000 cervical cancer cases and 1.18/100,000 cervical cancer deaths) and similar values for the 9-valent vaccine (5.47/100,000 cervical cancer cases predicted in the model versus 5.46/100,000 cases observed, and 1.51/100,000 cervical cancer deaths reported in the model versus 1.52/100,000 cervical cancer deaths observed).

Epidemiological/health outcomes

When compared with the 4-valent vaccine, gender-neutral vaccination with the 9-valent vaccine significantly reduced the prevalence of HPV types 16/18/31/33/45/52/58 (Fig. 1 and Fig. 2). The 9-valent vaccine reduced the incidence of HPV type 16/18/31/33/45/52/58-related HPV infection among males and females and CIN1, CIN2, CIN3, cervical cancer cases and cervical cancer deaths, over 100 years, as shown in Table 2 and from Figs. 3, 4, 5, 6 and 7. The 9-valent vaccine would prevent additional 12,702 cases of CIN1, 15,000 CIN2 and 13,862 CIN3 over 100 years in comparison to the 4-valent vaccine, which corresponds respectively to a 50, 48 and 36% cumulative incidence decrease (Table 2). The 9-valent vaccine is also associated with an additional cumulative decrease of cervical cancer incidence of 24% compared with the 4-valent vaccine, corresponding to additional 2979 cervical cancer cases and 741 deaths avoided in females over 100 years (Table 2).

Fig. 1
figure 1

Epidemiological impact of two vaccination strategies on HPV16/18/31/33/45/52/58 infection prevalence among females over 100 years

Full size image
Fig. 2
figure 2

Epidemiological impact of two vaccination strategies on HPV16/18/31/33/45/52/58 infection prevalence among males over 100 years

Full size image
Table 2 Disease events and deaths prevented with the two vaccination strategies associated with cervical screening
Full size table
Fig. 3
figure 3

Epidemiological impact of two vaccination strategies on HPV16/18/31/33/45/52/58 related incidence of CIN1 over 100 years

Full size image
Fig. 4
figure 4

Epidemiological impact of two vaccination strategies on HPV16/18/31/33/45/52/58 related incidence of CIN2 over 100 years

Full size image
Fig. 5
figure 5

Epidemiological impact of two vaccination strategies on HPV16/18/31/33/45/52/58 related incidence of CIN3 over 100 years

Full size image
Fig. 6
figure 6

Epidemiological impact of two vaccination strategies on HPV16/18/31/33/45/52/58 related cervical cancer incidence over 100 years

Full size image
Fig. 7
figure 7

Epidemiological impact of two vaccination strategies on HPV16/18/31/33/45/52/58 related cervical cancer deaths over 100 years

Full size image

The effect of 9-valent vaccination on HPV infection and precancerous cervical lesions incidence reduction is seen for an earlier time period than for cervical cancer. The reduction in incidence of HPV-related cancers and deaths from HPV-related cancers was more gradual, reflecting the fact that HPV-related cancers are diseases with slower progression (From Figs. 3, 4, 5, 6 and 7).

Sensitivity analyses

The results of the sensitivity analyses, illustrated by the cumulative reduction in HPV 16/18/31/33/45/52/58 related disease incidence cases, are presented in 8 and 9. The base case (80% of maximum VCR) over 100 years following vaccination is associated with a cumulative reduction of 12,702 CIN1, 15,000 CIN2, 13,862 CIN3, 2979 cervical cancers and 741 cervical cancer deaths compared with a 4-valent vaccination strategy. When other scenarios were tested with lower vaccination coverage rates, ranging from 30 to 60%, a substantial cumulative reduction for all tested outcomes was still observed (Fig. 8 and Fig. 9). Furthermore, when reducing the duration of protection from lifetime to 20 years, the results showed that the number of additional cancer cases and deaths prevented with the 9-valent vaccination versus the 4-valent vaccination was still significant, with additional 628 cervical cancer deaths avoided and approximately 2555 cervical cancer cases, 11,057 CIN1 cases, 13,051 CIN2 and 12,009 CIN3 cases avoided compared with the 4-valent vaccination (Fig. 8 and Fig. 9). The 20 years duration protection is a very conservative approach in light of the real world data available for the 4-valent vaccines already today [44]. The results of this conservative analysis tend to show that the epidemiological outcomes are not very sensitive to the duration of protection since this parameter has a limited impact on the model results.

Fig. 8
figure 8

Sensitivity analyses results – CIN1 and CIN2/3 incidence cases prevented with the two vaccination strategies

Full size image
Fig. 9
figure 9

Sensitivity analyses results – Cervical cancer incidence cases and deaths prevented with the two vaccination strategies

Full size image

Discussion

To our knowledge this is the first study assessing the epidemiological impact of a new vaccine strategy with the 9-valent vaccine in Switzerland. The analyses provide evidence for the epidemiological impact on cervical disease of a 9-valent vaccine in comparison to a 4-valent vaccine in a gender-neutral programme including girls and boys aged 11–26 years. The study was realized through collection and selection of the most relevant data to reflect the current clinical and epidemiological context in Switzerland, as closely as possible. The analyses show that 9-valent vaccination has a positive impact on the burden of HPV-related diseases in Switzerland. Assuming a maximum vaccination coverage of 80% in girls and boys aged 11 to 26 years, the 9-valent vaccination has the potential to additionally reduce, over 100-years following vaccination, cervical cancer incidence related to the 9 HPV types by 24%, cervical cancer deaths by 20%, CIN1 by 50%, CIN2 by 48% and CIN3 by 36%, respectively compared with a 4-valent vaccination strategy associated with cervical screening. Sensitivity analyses were conducted in order to test the impact of lower maximum vaccination coverage rates and a shorter duration of protection in the model outcomes. Results of these analyses are robust since a significant additional reduction in cervical cancer and precancerous lesions burden is still observed with the 9-valent vaccination compared with the 4-valent vaccination strategy.

Previous epidemiological models assessing gender-neutral HPV vaccination were already published in Europe. Ribassin-Majed et al. published in 2012 a deterministic transmission model comparing several vaccine scenarios in France. The authors observed that HPV vaccination may significantly decrease cervical cancer incidence with a stronger impact of gender-neutral vaccination considering the vaccine coverage in France in 2009 [45]. The epidemiological model from Choi et al. (2010) developed in the UK setting has shown that the 4-valent gender-neutral HPV vaccination was associated with an additional benefit in terms of reduction of cervical cancer and genital warts compared with a girls-only vaccination programme [46]. Other European epidemiological-focused published models in Germany [47], in Italy [48], in the Netherlands [49], and in Finland [50] have shown the positive epidemiological impact of gender-neutral HPV 2-valent and/or 4-valent vaccination. Besides, several cost-effectiveness studies recently published in Europe reported findings on the epidemiological impact of switching from a 4-valent vaccination to a 9-valent vaccination programme in a gender-neutral context [22,23,24]. For example, in Austria, a cost-effectiveness analysis conducted by Boiron et al. (2016) compared the epidemiological and economic outcomes of a 9-valent vaccination strategy versus a 4-valent vaccination programme in a gender-neutral context. This study showed that HPV-related diseases incidence could be reduced to a greater extent with the 9-valent vaccine (by 92% for HPV-related cervical cancers, by 96% for related CIN2/3 cases, by 83 and 76% for anal cancers respectively in females and males versus a screening-only strategy) compared with the 4-valent vaccine (incidence reduced by 75% for HPV-related cervical cancer, by 76% for related CIN2/3 cases, by 80 and 74% for anal cancers respectively in females and males versus a screening-only strategy). It represented 14,983 cases of CIN2/3 and 2544 cases of cervical cancer cases additionally prevented with the 9-valent vaccine over 100 years [22]. Similarly, a recent study examining the cost-effectiveness and epidemiological impact of gender-neutral vaccination with the 9-valent vaccine in Germany has also been published [23]. This analysis reported significant health benefit of a 9-valent vaccination compared with a 4-valent vaccination in terms of disease reduction: 24% reduction of cervical cancer incidence, 30 and 14% reduction of anal cancer incidence for males and females respectively, as well as over one million cases of genital warts avoided in 100 years [23]. Furthermore, an Italian study published by Mennini et al. in 2017 also assessed the cost-effectiveness of the 9-valent vaccine and reported the epidemiological impact of a switch from a 4-valent to a 9-valent vaccination [24]. The model results showed a significant diseases reduction in a gender-neutral programme with the 9-valent vaccine compared with the current vaccination programme with the 4-valent vaccine (reduction of 17% in the incidence of cervical cancer, 35 and 14% in anal cancer for males and females, and over a million cases of genital warts avoided after 100 years) [24].

While not the focus of this manuscript, the model used in this work also allowed us to estimate the cost-effectiveness of switching from the currently implemented HPV vaccination programme with the 4-valent vaccine to the 9-valent vaccine. Based on economic input parameters adapted to the Swiss setting [25] and considering the current private market price of the 4-valent vaccine, a price increase of 16 to 32% is estimated to be cost-effective based on ICER-thresholds used by NICE and WHO, respectively. The major strength is that the findings of this study are novel in the Swiss setting as previous epidemiological focused models and comparisons in specific European countries did not include the 9-valent versus the 4-valent vaccine among both males and females and were non-country-specific. Another strength of this analysis is the use of a dynamic model including herd immunity/protection effects.

The current analysis involves limitations. The first one is that some of the population-independent model inputs were not issued from Swiss-specific studies. However, to strengthen the study, all non-Swiss specific input values have been validated by expert opinion. A second limitation is that in order to simplify the calculations in the model, the epidemiological module simulated HPV types 6, 11, 16 and 18 separately, whereas the five additional types (31, 33, 45, 52 and 58) were combined into a single set of compartments. A third limitation was that this analysis only focused on the impact on cervical disease. While the main effect of a 9-valent HPV vaccination programme is predicted to occur for cervical disease, further incidence reduction is expected for other HPV-related diseases. The main difficulty encountered to adapt the model to the Swiss setting was to inform the VCR parameter. Indeed, a high heterogeneity of HPV vaccine coverage among cantons is observed in Switzerland. Uptake of HPV vaccination in 16 year old girls in Switzerland showed pronounced differences between different cantons ranging from 30 to 79% [21]. The maximum VCR used in the base case analysis was based on the assumption that the VCR in Switzerland would be higher than the observed rate in 2016 at the 9-valent vaccine launch and that it varied depending on gender and age of the vaccination (initial cohort or catch-up cohort) and the number of years after the 9-valent vaccine commercialization. To overcome the VCR variability between cantons and the results obtained in the base case analysis, sensitivity analyses on VCR parameter were conducted. The results showed that even with a low VCR of 30%, the 9-valent vaccine could greatly reduce the number of precancerous cervical lesions and cervical cancer cases and deaths and provided evidence that the 9-valent vaccine was still highly effective when compared with the 4-valent vaccine against HPV-related diseases. Riesen et al. published a dynamic transmission model to study the expected consequences of this spatial heterogeneity in vaccination uptake on the transmission and prevalence of HPV-16 in Switzerland. The authors concluded that spatial heterogeneity in HPV vaccination uptake is expected to diminish the effect of vaccination on HPV-16 prevalence, but the overall effect is small [51]. Thus, our findings are in line with the results by Riesen et al. [51].

It should be noted that several countries such as Ireland, Denmark and Japan experienced recently a sudden decline in VCR. For instance, in Ireland, the HPV vaccine uptake of the first dose dropped from 89.7% in 2014–2015 to 50% in 2016–2017 due to parental concerns about vaccine safety [52]. Although strong media campaigns are currently launched in Ireland to boost HPV vaccination, the former high levels of coverage rates are still not reached [52]. These recent events reveal the difficulty and time it takes to recover from a decline in VCR, reminding the necessity to support high HPV vaccination coverage rates in Swiss cantons.

A first model assessing the impact of HPV 4-valent vaccination has shown the effectiveness of HPV 4-valent vaccination in Switzerland by reducing HPV-related diseases burden in Switzerland [28]. An impact study was recently published and has demonstrated the short-term effectiveness of HPV vaccination with the 4-valent vaccine in a Swiss canton with a high VCR [53]. The data of our model suggest that replacing the current HPV 4-valent vaccination by the HPV 9-valent vaccination will further increase the impact on HPV-related diseases in Switzerland by reducing the burden of high-risk HPV genotypes other than HPV 16 or 18.

Real world data assessing the impact of HPV vaccination have demonstrated the public health impact on HPV infection, genital warts and cervical abnormalities. Garland et al. conducted a systematic review and have found reports for strong reductions in HPV types covered by the 4-valent HPV vaccine (up to 90%), for reductions of genital warts (up to 90%), for up to 45% reductions in low-grade cytological cervical abnormalities, and for up to 85% reductions in high-grade histologically proven cervical abnormalities. Even if the full public health potential of HPV vaccination is not yet realized, it can be easily expected that using 9-valent HPV universal vaccination will lead to an additional positive impact [44].

Conclusion

This study showed how the switch from a gender-neutral vaccination programme with the 4-valent vaccine to the 9-valent vaccine impacts cervical disease in the Swiss setting. The switch to the 9-valent vaccination would yield significant incremental epidemiological benefits by further reducing cervical cancer and precancerous lesions in Switzerland.

Availability of data and materials

Written requests can be sent to the corresponding author.

Model input data is publicly available [21, 22, 25,26,27, 29,30,31,32,33,34,35,36,37,38]. See methods section for details.

Abbreviations

BAG:

Bundesamt für Gesundheit/Federal Office of Public Health

BFS:

Bundesamt für Statistik/Federal Office of Statistics

CIN:

Cervical Intraepithelial Neoplasia

EKIF:

Federal Vaccination Commission

HPV:

Human papillomavirus

ICER:

Incremental cost effectiveness ratio

NATSAL:

National Survey of Sexual Attitudes and Lifestyles

NICER:

National Institute for Cancer Epidemiology and Registration

RRP:

Recurrent respiratory papillomatosis

US:

United States

VCR:

Vaccination coverage rate

References

  1. Giuliano AR, Nyitray AG, Kreimer AR, Pierce Campbell CM, Goodman MT, Sudenga SL, et al. EUROGIN 2014 roadmap: differences in human papillomavirus infection natural history, transmission and human papillomavirus-related cancer incidence by gender and anatomic site of infection. Int J Cancer. 2015;136(12):2752–60.

    Article  CAS  PubMed  Google Scholar 

  2. WHO. Human papillomavirus (HPV) and cervical cancer; 2016. Available from: http://www.who.int/mediacentre/factsheets/fs380/en/.

    Google Scholar 

  3. Serrano B, Brotons M, Bosch FX, Bruni L. Epidemiology and burden of HPV-related disease. Best Pract Res Clin Obstet Gynaecol. 2018;47:14–26.

    Article  PubMed  Google Scholar 

  4. Hartwig S, St Guily JL, Dominiak-Felden G, Alemany L, de Sanjose S. Estimation of the overall burden of cancers, precancerous lesions, and genital warts attributable to 9-valent HPV vaccine types in women and men in Europe. Infect Agent Cancer. 2017;12:19.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  5. FOPH. Human papilloma virus: Federal Public Health Office (BAG); 2018. Available from: https://www.bag.admin.ch/bag/en/home/themen/mensch-gesundheit/uebertragbare-krankheiten/infektionskrankheiten-a-z/hpv.html.

    Google Scholar 

  6. National Institute for Cancer EaRN. Cervical cancer annual incidence rates; 2013. Available from: http://www.nicer.org/en/statistics-atlas/cancer-incidence/.

    Google Scholar 

  7. Hartwig S, Baldauf JJ, Dominiak-Felden G. Estimation of the epidemiological burden of HPV-related cancers, precancerous lesions, and genital warts in women and men in Europe: potential additional benefit of a nine-valent compared to the quadrivalent HPV vaccine; 2015.

    Google Scholar 

  8. FOPH. Bulletin 6 du 6 février 2017 - les génotypes des papillomavirus humains dans les lésions précancéreuses et les cancers du col de l’utérus en Suisse au début des programmes cantonaux de vaccination: l’étude CIN3+plus: Federal Public Health Office (BAG); 2017.

  9. FOPH. Bulletin 3 du 15 janvier 2018 - La vaccination contre les HPV : efficace et sûre: Federal Public Health Office (BAG); 2018.

  10. Frey Terri B, Petignat P, Jacot-Guillarmod M, Mueller M, Fher M, Kind A. Avis d'expert no 50 - Recommandations pour la prévention du cancer du col de l'utérusSociété Suisse de Gynécologie et d'Obstétrique; 2018.

    Google Scholar 

  11. Swissmedic. Gardasil®- product information; 2016. Available from: http://www.swissmedicinfo.ch/default.aspx.

    Google Scholar 

  12. Cervarix - Informations professionnelles du Compendium suisse des médicaments 2016 [Available from: https://compendium.ch/mpro/mnr/21274/pdf/fr#page1.

  13. IMS. Market shares of Gardasil® and Cervarix® - MAT June 2017; 2017.

    Google Scholar 

  14. Giuliano AR, Joura EA, Garland SM, Huh WK, Iversen OE, Kjaer SK, et al. Nine-valent HPV vaccine efficacy against related diseases and definitive therapy: comparison with historic placebo population. Gynecol Oncol. 2019;154(1):110–7.

    Article  CAS  PubMed  Google Scholar 

  15. FOPH. Vaccination against HPV vaccines : recommendation of complementary vaccination for boys and young men aged 11–26 years; 2015. Available from: www.bag.admin.ch.

    Google Scholar 

  16. Takla A, Pm W, Harder T, Meerpohl JJ, Röbl-Mathiue M, Terhardt M, et al. Background paper for the recommendation of HPV vaccination for boys in Germany. Bundesgesundheitsbl. 2018;61:1170–86.

    Article  Google Scholar 

  17. Joint Committe on Vaccination and Immunisation. Statement on HPV vaccination: Joint Committe on Vaccination and Immunisation; 2018. July 2018.

  18. Joint Committe on Vaccination and Immunisation. Minute of the meeting on 06 June 2018Joint Committe on vaccination and immunisation,; 2018 06 June; 2018.

    Google Scholar 

  19. Health Information and Quality Authority. Report on the results of the public consultation on the draft health technology assessment (HTA) of extending the national immunisation schedule to include HPV vaccination of boysHealth Information and Quality Authority,; 2018 04 December; 2018.

    Google Scholar 

  20. Wymann MN, Zographos AS, Altpeter E, Spicher VM, Low N, Mausezahl-Feuz M. Human papillomavirus vaccine uptake in adolescence and adherence to cervical cancer screening in Switzerland: a national cross-sectional survey. Int J Public Health. 2018;63(1):105–14.

    Article  PubMed  Google Scholar 

  21. Federal Office of statistics. National and cantonal vaccination coverage rates in Switzerland from 1999 to 2016; 2016. [Available from: https://www.bag.admin.ch/bag/fr/home/themen/mensch-gesundheit/uebertragbare-krankheiten/impfungen-prophylaxe/informationen-fachleute-gesundheitspersonal/durchimpfung.html.

    Google Scholar 

  22. Boiron L, Joura E, Largeron N, Prager B, Uhart M. Estimating the cost-effectiveness profile of a universal vaccination programme with a nine-valent HPV vaccine in Austria. BMC Infect Dis. 2016;16:153.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  23. Largeron N, Petry KU, Jacob J, Bianic F, Anger D, Uhart M. An estimate of the public health impact and cost-effectiveness of universal vaccination with a 9-valent HPV vaccine in Germany. Expert Rev Pharmacoecon Outcomes Res. 2017;17(1):85–98.

    Article  PubMed  Google Scholar 

  24. Mennini FS, Bonanni P, Bianic F, de Waure C, Baio G, Plazzotta G, et al. Cost-effectiveness analysis of the nine-valent HPV vaccine in Italy. Cost Eff Resour Alloc. 2017;15:11.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Federal Office of Statistics. Population size in Switzerland in 2015; 2016. [Available from: https://www.bfs.admin.ch/bfs/fr/home/statistiques/population/effectif-evolution/repartition-territoriale.assetdetail.290789.html.

    Google Scholar 

  26. Mercer CH, Tanton C, Prah P, Erens B, Sonnenberg P, Clifton S, et al. Changes in sexual attitudes and lifestyles in Britain through the life course and over time: findings from the National Surveys of sexual attitudes and lifestyles (Natsal). Lancet. 2013;382(9907):1781–94.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Elbasha EH, Dasbach EJ. An integrated economic evaluation and HPV disease transmission models - technical report accompanying the manuscript impact of vaccinating boys and men against HPV in the Inited states. Vaccine. 2010;28(42):6858–67.

    Article  PubMed  Google Scholar 

  28. Szucs TD, Largeron N, Dedes KJ, Rafia R, Benard S. Cost-effectiveness analysis of adding a quadrivalent HPV vaccine to the cervical cancer screening programme in Switzerland. Curr Med Res Opin. 2008;24(5):1473–83.

    Article  PubMed  Google Scholar 

  29. Burton-Jeangros C, Cullati S, Manor O, Courvoisier DS, Bouchardy C, Guessous I. Cervical cancer screening in Switzerland: cross-sectional trends (1992-2012) in social inequalities. Eur J Pub Health. 2017;27(1):167–73.

    Google Scholar 

  30. Elbasha EH, Dasbach EJ. Impact of vaccinating boys and men against HPV in the United States. Vaccine. 2010;28(42):6858–67.

    Article  PubMed  Google Scholar 

  31. Merck. Merck's Investigational 9-valent HPV Vaccine, V503, Prevented 97 Percent of Cervical, Vaginal and Vulvar Pre-cancers Caused by Five Additional HPV Types, in Phase III Study 2015 [updated 2015. Available from: http://www.mercknewsroom.com/news-release/research-and-development-news/mercks-investigational-9-valent-hpv-vaccine-v503-prevente.

  32. EUROCARE. EUROCARE 5 survival analysis 2000-2007 2016 [Available from: http://www.eurocare.it/Publications/tabid/61/Default.aspx.

  33. Giuliano AR, Palefsky JM, Goldstone S, Moreira ED Jr, Penny ME, Aranda C, et al. Efficacy of quadrivalent HPV vaccine against HPV infection and disease in males. N Engl J Med. 2011;364(5):401–11.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  34. Ault KA. Effect of prophylactic human papillomavirus L1 virus-like-particle vaccine on risk of cervical intraepithelial neoplasia grade 2, grade 3, and adenocarcinoma in situ: a combined analysis of four randomised clinical trials. Lancet. 2007;369(9576):1861–8.

    Article  PubMed  CAS  Google Scholar 

  35. Garland SM, Hernandez-Avila M, Wheeler CM, Perez G, Harper DM, Leodolter S, et al. Quadrivalent vaccine against human papillomavirus to prevent anogenital diseases. N Engl J Med. 2007;356(19):1928–43.

    Article  CAS  PubMed  Google Scholar 

  36. Joura EA, Giuliano AR, Iversen OE, Bouchard C, Mao C, Mehlsen J, et al. A 9-valent HPV vaccine against infection and intraepithelial neoplasia in women. N Engl J Med. 2015;372(8):711–23.

    Article  CAS  PubMed  Google Scholar 

  37. Joura EA, Leodolter S, Hernandez-Avila M, Wheeler CM, Perez G, Koutsky LA, et al. Efficacy of a quadrivalent prophylactic human papillomavirus (types 6, 11, 16, and 18) L1 virus-like-particle vaccine against high-grade vulval and vaginal lesions: a combined analysis of three randomised clinical trials. Lancet. 2007;369(9574):1693–702.

    Article  CAS  PubMed  Google Scholar 

  38. Palefsky JM, Giuliano AR, Goldstone S, Moreira ED Jr, Aranda C, Jessen H, et al. HPV vaccine against anal HPV infection and anal intraepithelial neoplasia. N Engl J Med. 2011;365(17):1576–85.

    Article  CAS  PubMed  Google Scholar 

  39. Cervical cancer annual mortality rates: National Institute for Cancer, Epidemiology and Registration (NICER); 2013 [Available from: http://www.nicer.org/en/statistics-atlas/cancer-mortality/.

  40. Cervical cancer annual incidence rates: National Institute for Cancer, Epidemiology and Registration (NICER); 2013 [Available from: http://www.nicer.org/en/statistics-atlas/cancer-incidence/.

  41. Vaccination against HPV vaccines : recommendation of complementary vaccination for boys and young men aged 11–26 years. Federal Public Health Office (BAG); 2015.

  42. Hartwig S, Baldauf JJ, Dominiak-Felden G, Simondon F, Alemany L, de Sanjosé S, et al. Estimation of the epidemiological burden of HPV-related cancers, precancerous lesions, and genital warts in women and men in Europe: potential additional benefit of a nine-valent second generation HPV vaccine compared to first generation HPV vaccines. Papillomavirus Research. 2015;1:90–100.

    Article  PubMed Central  Google Scholar 

  43. Hartwig S, Syrjanen S, Dominiak-Felden G, Brotons M, Castellsague X. Estimation of the epidemiological burden of human papillomavirus-related cancers and non-malignant diseases in men in Europe: a review. BMCCancer. 2012;12:30.

    Google Scholar 

  44. Garland SM, Kjaer SK, Munoz N, Block SL, Brown DR, DiNubile MJ, et al. Impact and effectiveness of the Quadrivalent human papillomavirus vaccine: a systematic review of 10 years of real-world experience. Clin Infect Dis. 2016;63(4):519–27.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. Ribassin-Majed L, Lounes R, Clemencon S. Efficacy of vaccination against HPV infections to prevent cervical cancer in France: present assessment and pathways to improve vaccination policies. PLoS One. 2012;7(3):e32251.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Choi YH, Jit M, Gay N, Cox A, Garnett GP, Edmunds WJ. Transmission dynamic modelling of the impact of human papillomavirus vaccination in the United Kingdom. Vaccine. 2010;28(24):4091–102.

    Article  PubMed  Google Scholar 

  47. Horn J, Damm O, Kretzschmar ME, Delere Y, Wichmann O, Kaufmann AM, et al. Estimating the long-term effects of HPV vaccination in Germany. Vaccine. 2013;31(19):2372–80.

    Article  CAS  PubMed  Google Scholar 

  48. Baussano I, Dillner J, Lazzarato F, Ronco G, Franceschi S. Upscaling human papillomavirus vaccination in high-income countries: impact assessment based on transmission model. Infect Agent Cancer. 2014;9(1):4.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Matthijsse SM, Hontelez JA, Naber SK, Rozemeijer K, de Kok IM, Bakker R, et al. Public health benefits of routine human papillomavirus vaccination for adults in the Netherlands: a mathematical modeling study. J Infect Dis. 2016;214(6):854–61.

    Article  PubMed  Google Scholar 

  50. French KM, Barnabas RV, Lehtinen M, Kontula O, Pukkala E, Dillner J, et al. Strategies for the introduction of human papillomavirus vaccination: modelling the optimum age- and sex-specific pattern of vaccination in Finland. Br J Cancer. 2007;96(3):514–8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Riesen M, Garcia V, Low N, Althaus CL. Modeling the consequences of regional heterogeneity in human papillomavirus (HPV) vaccination uptake on transmission in Switzerland. Vaccine. 2017;35(52):7312–21.

    Article  PubMed  Google Scholar 

  52. Corcoran BCA, Barett T. Rapid response to HPV vaccination crisis in Ireland. Lancet. 2018.

  53. Jacot-Guillarmod M, Pasquier J, Greub G, Bongiovanni M, Achtari C, Sahli R. Impact of HPV vaccination with Gardasil(R) in Switzerland. BMC Infect Dis. 2017;17(1):790.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Caroline Guilmet and Claire Okaïs from ICON for support in manuscript writing. The abstract of this work has previously been presented at the EUROGIN congress 2018 (Epidemiologic impact of a nonavalent GNV programme in Switzerland, December 2018, presented at EUROGIN 2018, Lisbon, Portugal).

Funding

This study was funded by Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., Kenilworth, NJ, USA. The funder contributed to the design of the study, collection, analysis, interpretation of data as well as the writing and revision of the manuscript.

Author information

Authors and Affiliations

  1. Department of Gynaecology and Gynaecological Oncology, University Hospital Basel, Spitalstrasse 21, CH-4056, Basel, Switzerland

    André B. Kind

  2. Center for Observational and Real-World Evidence (CORE), Merck & Co., Inc, Kenilworth, NJ, 07033, USA

    Andrew Pavelyev & Smita Kothari

  3. HCL America, Inc., Sunnyvale, CA, USA

    Andrew Pavelyev

  4. ICON plc, 55 avenue des Champs Pierreux, 92000, Nanterre, France

    Nadia El Mouaddin

  5. ICON plc, 27 rue de la Villette, 69003, Lyon, France

    Aurélie Schmidt

  6. Outcomes Research, MSD, 162 Avenue Jean Jaurès, 69007, Lyon, France

    Edith Morais

  7. Market Access, MSD, Werftestrasse 4, CH-6005, Luzern, Switzerland

    Patrik Guggisberg

  8. Medical Affairs, MSD, Werftestrasse 4, CH-6005, Luzern, Switzerland

    Florian Lienert

Authors
  1. André B. Kind
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew Pavelyev
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Smita Kothari
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nadia El Mouaddin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aurélie Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Edith Morais
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Patrik Guggisberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Florian Lienert
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AK: provided model input, interpreted the data, and revised the manuscript. AP: ran the model, produced the analyses, interpreted the data, and produced the analyses. AS: was involved in the analyses and wrote the manuscript. SK: was involved in the analyses and revised the manuscript. NE: ran the model, produced the analyses, was involved in the analyses, and wrote the manuscript. EM: was involved in the analyses, interpreted the data, and revised the manuscript. PG: contributed to model input, interpreted the data, and revised the manuscript. FL: contributed to model input, interpreted the data, and revised the manuscript. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Florian Lienert.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

SK, EM, PG, and FL are employees of MSD, the 9-valent vaccine manufacturer. AS, AP and NE were working on this study on behalf of MSD, the 9-valent vaccine manufacturer. AK has served on a scientific advisory board of MSD, the 9-valent vaccine manufacturer.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kind, A., Pavelyev, A., Kothari, S. et al. Assessing the epidemiological impact on cervical cancer of switching from 4-valent to 9-valent HPV vaccine within a gender-neutral vaccination programme in Switzerland. BMC Public Health 20, 671 (2020). https://doi.org/10.1186/s12889-020-08840-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12889-020-08840-0

Keywords

BMC Public Health

ISSN: 1471-2458

Contact us