Feasibility and impact of providing feedback to vaccinating medical clinics: evaluating a public health intervention
BMC Public Health volume 10, Article number: 750 (2010)
Vaccine coverage (VC) at a given age is a widely-used indicator for measuring the performance of vaccination programs. However, there is increasing data suggesting that measuring delays in administering vaccines complements the measure of VC. Providing feedback to vaccinators is recognized as an effective strategy for improving vaccine coverage, but its implementation has not been widely documented in Canada. The objective of this study was to evaluate the feasibility of providing personalized feedback to vaccinators and its impact on vaccination delays (VD).
In April and May 2008, a one-hour personalized feedback session was provided to health professionals in vaccinating medical clinics in the Quebec City region. VD for vaccines administered at two and twelve months of age were presented. Data from the regional vaccination registry were analysed for participating clinics. Two 12-month periods before and after the intervention were compared, namely from April 1st, 2007 to March 31st, 2008 and from June 1st, 2008 to May 31st, 2009.
Ten medical clinics out of the twelve approached (83%), representing more than 2500 vaccinated children, participated in the project. Preparing and conducting the feedback involved 20 hours of work and expenses of $1000 per clinic. Based on a delay of one month, 94% of first doses of DTaP-Polio-Hib and 77% of meningococcal vaccine doses respected the vaccination schedule both before and after the intervention. Following the feedback, respect of the vaccination schedule increased for vaccines planned at 12 months for the four clinics that had modified their vaccination practices related to multiple injections (depending on the clinic, VD decreased by 24.4%, 32.0%, 40.2% and 44.6% respectively, p < 0.001 for all comparisons).
The present study shows that it is feasible to provide personalized feedback to vaccinating clinics. While it may have encouraged positive changes in practice concerning multiple injections, this intervention on its own did not impact vaccination delays of the clinics visited. It is possible that feedback integrated into other types of effective interventions and sustained over time may have more impact on VD.
Child vaccination is one of the most effective interventions in public health [1, 2]. Vaccine coverage (VC) at a given age is a widely-used indicator for measuring the performance of vaccination programs . However, there is increasing data suggesting that measuring delays in age-appropriate vaccination complements the measure of VC [4–6]. Vaccination delays (VD) can, independently of VC, increase the burden of certain childhood diseases such as pertussis and measles [7, 8]. Moreover, a delay in the administration of vaccines scheduled for the age of two months is associated with poor VC at two years [3, 9, 10].
Across Canada, a consensus has been reached on the definition of VD, namely one month after the due date on the vaccination schedule . This definition is in line with the principal studies on the subject [4, 6, 12–15]. However, since 2006, the Quebec Ministry of Health and Social Services has compiled VD noted in Health and Social Service Centres (Centres de santé et de services sociaux or CSSS) based on a delay of one week following the recommended age . This tracking of VD is not carried out in private medical clinics, which administer about half of the vaccines for infants in the province of Quebec [3, 17].
In 2008, in Quebec, using a one-month delay definition, only 21% of children received all the vaccines scheduled before the age of two years within the recommended period .
Providing feedback to vaccinators is one of the effective strategies for improving VC [18–24]. It involves presenting clinicians with their performance data, with or without recommendations . This intervention is carried out following data collection on clinicians' performance (audit). Certain studies have shown that feedback may help reduce VD and missed vaccination opportunities [18, 26]. On the other hand, the impact and feasibility of providing feedback to vaccinating medical clinics have not been documented extensively in a Canadian context [27, 28].
The main objective of this study was to evaluate the feasibility and the impact on VD of providing personalized feedback to medical clinics that vaccinate infants in the Quebec City region.
The twelve medical clinics that had administered the largest number of doses of DTaP-Polio-Hib vaccine in 2007 in the Quebec City region were approached to participate in the feedback project. The vaccination registry (VAXIN), which compiles all doses administered in the Quebec City region, enabled to identify these clinics and to evaluate vaccination delays.
In April and May 2008, a one-hour feedback session, led by a physician and a public health nurse, was carried out with the physicians, nurses and secretaries in each participating medical clinic. This feedback dealt with VD for infants at the clinic for the year 2007. Data on the proportion of doses administered without delay were presented for the first doses of three vaccines (DTaP-Polio-Hib, pneumococcal and meningococcal). Vaccination delays for each clinic were presented both in terms of the Quebec standard (one week) and the proposed Canadian standard (one month). Graphs showing the cumulative percentage of children vaccinated according to age were also presented for vaccines scheduled at 2 and 12 months, including measles, mumps and rubella vaccine (MMR). During preparation of the feedback, it became clear that certain clinics were not administering vaccines scheduled at one year during a single visit. Consequently, information on the importance of multiple injections was transmitted to these medical clinics. Ethical approval was obtained from the Comité d'éthique de la recherche du CHUQ - Centre hospitalier de l'Université Laval (project C10-11-101).
At the end of each feedback session, a questionnaire on the organizational characteristics of the clinic in terms of vaccination was completed on site by a nurse or a secretary. One year following the feedback session (July 2009), the same questionnaire was completed by telephone and, when possible, with the same person.
The data related to the feasibility of the intervention, i.e., the material, human and financial resources required, were compiled.
Analyses of vaccination delays
In order to measure the impact of the feedback on delays associated with vaccines (1st dose of DTaP-Polio-Hib, 1st dose of pneumococcal, meningococcal and 1st dose of MMR), data from the regional vaccination registry were analyzed for the participating clinics. Two 12-month periods before and after the intervention were compared, namely from April 1st, 2007 to March 31st, 2008 and from June 1st, 2008 to May 31st, 2009. Two indicators of delay were considered: one week and one month after the due date on the vaccination schedule. Proportion comparisons were carried out using the chi-square test. The distributions of ages at vaccination, before and after the intervention, were compared using the Wilcoxon test. The analyses were carried out using the SAS 9.1 software (SAS Institute, Cary, NC).
A total of ten clinics (83%), representing more than 2500 vaccinated children, agreed to participate in the feedback project. Out of the 206 physicians, nurses and secretaries working in these ten clinics, 106 participated in the feedback process (51%).
Organizational characteristics of the medical clinics
Table 1 summarizes the organizational characteristics of the clinics visited and the changes observed between the ends of each of the two observation periods. Among other characteristics, it was noted that four clinics changed their practices concerning multiple injections. For two of these four clinics, responsibilities concerning vaccination were given to nurses. Since 2002, in the province of Quebec, nurses have the possibility to vaccinate as set out in the Public Health Act .
During the second observation period, 94% of the first doses of DTaP-Polio-Hib and pneumococcal vaccines were received within the one-month delay (Table 2). This proportion was 77% for meningococcal vaccine and 75% for MMR (Table 3). When a delay of one week after the scheduled vaccination date was applied, the proportion of doses received without delay was 56% for DTaP-Polio-Hib and pneumococcal vaccines, 34% for meningococcal vaccine and 33% for MMR.
Few changes were observed before and after the feedback (Tables 2 and 3). Using the one-month delay, no statistically significant difference in VD was found between the two periods. Using the one-week delay, a statistically significant increase in VD was observed for DTaP-Polio-Hib (8.9%, p < 0.001), pneumococcal (9.0%, p < 0.001) and meningococcal (2.7%, p = 0.036) vaccines.
The median age at vaccination increased in a statistically significant manner for DTaP-Polio-Hib and pneumococcal vaccines (+ 2 days, p < 0.0001). For all vaccines, the difference in median age at vaccination before and after the feedback was nevertheless very small, consistently less than 3 days (Table 4).
As can be seen in Figure 1, age at vaccination for DTaP-Polio-Hib and meningococcal vaccines for the 2008-2009 period was comparable to that for the 2007-2008 period, with both curves being superimposed.
For the four clinics that began the practice of multiple injections in 2008-2009, a significant decrease in VD was observed for vaccines scheduled at 12 months. For the third dose of the pneumococcal vaccine (given at 12 months), the proportion of infants immunized within a one-month delay increased from 19.6% to 51.6% (p < 0,001) for clinic 10 and from 29.3% to 73.9% (p < 0,001) for clinic 09. The proportion of infants immunized without delay for the first dose of MMR increased from 27.4% to 67.6% (p < 0,001) for clinic 05. Finally, the proportion of infants immunized without delay for the meningococcal vaccine increased from 56.5% to 80.9% (p < 0,001) for clinic 06. Figure 2 (a and b) illustrates this situation for two of the four clinics.
The preparation and completion of the feedback sessions required, in terms of human resources, the contribution of a physician (42 hours), two nurses (48 hours), a statistician (42 hours) and a research officer (74 hours). The cost of preparing and carrying out the meetings in the ten clinics (professional hours, materials, travel and cold meals served) was estimated at $10,000.
The feedback process conducted in 10 of the 12 main vaccinating clinics in the Quebec City region did not decrease overall vaccination delays. A high proportion of the DTaP-Polio-Hib and pneumococcal vaccines were administered before the age of three months (94%), both before and after the intervention. The proportion of doses administered without delay was lower for meningococcal (77%) and MMR (73-75%) vaccines.
Vaccination delays observed in this study are similar or compare favourably with VD documented in other countries for MMR [13, 14] or for vaccines containing certain elements of DTaP-Polio-Hib [30, 31]. Nevertheless caution must be exercised in comparing VD data given the different methodologies employed.
Several reasons could explain the low impact of the intervention conducted herein. Firstly, this was an isolated intervention not combined with other types of interventional activities. Feedback that is integrated into other activities and sustained over time, such as that inspired by the Assessment, Feedback, Incentives and Exchange (AFIX) strategy, appears to be more likely to have an impact on vaccination [20, 23, 32]. Moreover, an increase in the number of doses administered in 2008-2009 was noted compared to 2007-2008. The pressure represented by this increased demand for vaccination could have contributed to the low impact of the feedback. As well, only half of the health professionals involved in vaccination in the clinics that were visited participated in the feedback process. The impact analyses, on the other hand, dealt with the totality of vaccination acts in the participating clinics, since the registry does not include a specific code for each vaccinator. Finally, VD presented were relatively low for some clinics and may not have generated sufficient motivation to change current practices. However, the feedback provided could have influenced more specific problematic practices resulting in high VD.
It is possible that the feedback was the cause of the decrease in VD observed for the four clinics that modified their practices concerning multiple injections. It is also possible that the introduction of the combined measles, mumps, rubella and varicella vaccine (MMR-V) administered at one year of age as well as the involvement of nurses in vaccination contributed to these changes. Two of the four clinics involved hired nurses or gave nurses additional responsibilities following the feedback. Other studies suggest that the involvement of nurses favours adherence to recommendations regarding vaccination . These results are important, since the practice of multiple injections is a factor that facilitates complete immunization . However, these modifications did not appear to have an impact on global analyses. A possible hypothesis is that the decrease in VD was observed in four of the smallest clinics that only gave a limited number of vaccine doses, therefore having a limited weight compared to larger clinics.
A relatively wide consensus exists concerning the definition of VD, namely one month after the date on the immunization schedule . In the present study, the results using this standard did not suggest any deterioration or improvement in VD after feedback. When using a one-week standard, we observed an increase in VD for the first DTaP-Polio-Hib vaccine, for the first pneumococcal vaccine and for the meningococcal vaccine. However, these increases in VD correspond to a mean immunization time delayed by only one or two days in the year following the feedback, a difference not clinically significant. It is possible that using a one-week definition for VD may be too sensitive to very minor changes in immunization practices and lead to misinterpretations. Furthermore, using a short one-week standard for VD could demoralize clinicians because it is very difficult to reach. This issue was reported by some participants during the feedback sessions.
In Quebec, the Ministry of Health and Social Services tracks the vaccination delays associated with DTaP-Polio-Hib, pneumococcal and meningococcal vaccines. Strict surveillance of VD for DTaP-Polio-Hib vaccine is essential, as delays in its administration are associated with incomplete VC . Since the VD for pneumococcal vaccine are almost identical to that for DTaP-Polio-Hib, and since it is scheduled for the same visit, the benefits associated with its surveillance appear rather limited.
A number of limitations of this study should be mentioned. A high proportion of vaccination acts are recorded in the vaccination registry used (VAXIN) . However, when children move to the Quebec City region, it is possible that vaccines given previously in other regions may not be systematically reported and recorded. This phenomenon is apt to lead to an overestimation of VD, but should be consistent over time. The fact that the VD determined by this study are close to the data reported in provincial studies is nevertheless reassuring .
The questionnaire was completed in person after the first observation period and by telephone after the second observation period. As well, the same person was reached in only five of the ten clinics. An effort was made to limit the impact of this latter factor by formulating the questions in the same manner.
Finally, the use of a control group would have allowed for the impact of the feedback to be better identified. However, such a strategy would have been extremely difficult to apply since a large majority of doses are administered by a small number of clinics in the Quebec City region and the clinics that vaccinate the most were the ones that participated in the study. Nevertheless, we compared the VD observed during the two periods for those clinics that did not participate in the project and the same tendencies were noted.
While the feedback that was provided did not lead to a decrease in VD, it nevertheless appeared to have facilitated certain positive changes in practices concerning multiple injections. It is possible that feedback integrated into other types of effective interventions, sustained over time or giving certain responsibilities to vaccinators may have more impact in decreasing VD. These measures, however, would require additional resources. Efforts aimed at reducing the burden related to delayed immunization and at improving VC should be pursued in order to protect the population against diseases that can be avoided by vaccination.
Diphtheria, tetanus toxoid, acellular pertussis, inactivated poliovirus and Haemophilus influenzae type b vaccine
Measles, mumps and rubella vaccine
Measles, mumps, rubella and varicella vaccine
Bos E, Batson A: Using immunization coverage rates for monitoring health sector performance: Measurement and interpretation issues. 2000, Washington DC: Human development network, The World Bank, 1-21.
Comité consultatif national sur l'immunisation (Ed.): Guide canadien d'immunisation. 2006, Ottawa: Agence de la santé publique du Canada, 410-7
Boulianne N, Bradet R, Audet D, Deceuninck G: Enquête sur la couverture vaccinale des enfants de 1 an et 2 ans au Québec en 2008. 2009, Québec: Institut national de santé publique du Québec, 1-205.
Dombrowski KJ, Lantz PM, Freed GL: The need for surveillance of delay in age-appropriate immunization. Am J Prev Med. 2002, 23 (1): 36-42. 10.1016/S0749-3797(02)00442-7.
Luman ET, McCauley MM, Stokley S, Chu SY, Pickering LK: Timeliness of childhood immunizations. Pediatrics. 2002, 110 (5): 935-939. 10.1542/peds.110.5.935.
Akmatov MK, Kretzschmar M, Kramer A, Mikolajczyk RT: Timeliness of vaccination and its effects on fraction of vaccinated population. Vaccine. 2008, 26 (31): 3805-3811. 10.1016/j.vaccine.2008.05.031.
Shinall MC, Peters TR, Zhu Y, Chen Q, Poehling KA: Potential impact of acceleration of the pertussis vaccine primary series for infants. Pediatrics. 2008, 122 (5): 1021-1026. 10.1542/peds.2007-3025.
The National Vaccine Advisory Committee: The measles epidemic. The problems, barriers, and recommendations. JAMA. 1991, 266 (11): 1547-1552. 10.1001/jama.266.11.1547.
Dietz VJ, Stevenson J, Zell ER, Cochi S, Hadler S, Eddins D: Potential impact on vaccination coverage levels by administering vaccines simultaneously and reducing dropout rates. Arch Pediatr Adolesc Med. 1994, 148 (9): 943-948.
Wood D, Pereyra M, Halfon N, Hamlin J, Grabowsky M: Vaccination levels in Los Angeles public health centers: The contribution of missed opportunities to vaccinate and other factors. Am J Public Health. 1995, 85 (6): 850-853. 10.2105/AJPH.85.6.850.
Canadian Immunization Registry network Task Group, Boulianne N, Hemon Y-A, Mawhinney T, Strong D, Gemmill I, Dobson S, Sartison E, Sargent M, Naus M, Tuchscherer R, Craig E, Watkins K, Schouten H: National eligible, due and overdue guidelines for immunization registries: draft recommendations from the Canadian Immunization Registry Network, Data Standards Task Group. Can Commun Dis Rep. 2004, 30 (6): 53-59.
Luman ET, Barker LE, Shaw KM, McCauley MM, Buehler JW, Pickering LK: Timeliness of childhood vaccinations in the United States: days undervaccinated and number of vaccines delayed. JAMA. 2005, 293 (10): 1204-1211. 10.1001/jama.293.10.1204.
Hull BP, McIntyre PB: Timeliness of childhood immunisation in Australia. Vaccine. 2006, 24 (20): 4403-4408. 10.1016/j.vaccine.2006.02.049.
Dayan GH, Shaw KM, Baughman AL, Orellana LC, Forlenza R, Ellis A, Chaui J, Kaplan S, Strebel P: Assessment of delay in age-appropriate vaccination using survival analysis. Am J Epidemiol. 2006, 163 (6): 561-570. 10.1093/aje/kwj074.
Grant CC, Roberts M, Scragg R, Stewart J, Lennon D, Kivell D, Ford R, Menzies R: Delayed immunisation and risk of pertussis in infants: unmatched case-control study. BMJ. 2003, 326 (7394): 852-853. 10.1136/bmj.326.7394.852.
Ministère de la Santé et des Services sociaux: Ententes de gestion, fiche descriptive des indicateurs - Fiche 1.1 FGJ. 2006
Sauvageau C, Trépanier J-P, Guay M, Landry M, Ben-Yedder N, Laflamme M, Minh Man Nguyen M, Weil A, Hudson P, Royer L, Ouakki M, Pelletier A, Boulianne N: Vaccination en clinique médicale: y a-t-il des délais?. 2006, Montréal. Journées annuelles de santé publique, 23-27.
LeBaron CW, Chaney M, Baughman AL, Dini EF, Maes E, Dietz V, Bernier R: Impact of measurement and feedback on vaccination coverage in public clinics, 1988-1994. JAMA. 1997, 277 (8): 631-635. 10.1001/jama.277.8.631.
Briss PA, Rodewald LE, Hinman AR, Shefer AM, Strikas RA, Bernier RR, Carande-Kulis VG, Yusuf HR, Ndiaye SM, Williams SM: Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med. 2000, 18 (1S): 97-140. 10.1016/S0749-3797(99)00118-X.
LeBaron CW, Mercer JT, Massoudi MS, Dini E, Stevenson J, Fischer WM, Loy H, Quick LS, Warming JC, Tormey P, DesVignes-Kendrick M: Changes in clinic vaccination coverage after institution of measurement and feedback in 4 states and 2 cities. Arch Pediatr Adolesc Med. 1999, 153 (8): 879-886.
Bordley WC, Chelminski A, Margolis PA, Kraus R, Szilagyi PG, Vann JJ: The effect of audit and feedback on immunization delivery: a systematic review. Am J Prev Med. 2000, 18 (4): 343-350. 10.1016/S0749-3797(00)00126-4.
Fairbrother G, Hanson KL, Friedman S, Butts GC: The impact of physician bonuses, enhanced fees, and feedback on childhood immunization coverage rates. Am J Public Health. 1999, 89 (2): 171-175. 10.2105/AJPH.89.2.171.
Melinkovich P, Hammer A, Staudenmaier A, Berg M: Improving pediatric immunization rates in a safety-net delivery system. Jt Comm J Qual Patient Saf. 2007, 33 (4): 205-210.
Zara S, Briss PA, Harris KW: The Guide to Community Preventive Services - What Works to Promote Health?. Edited by: Stephanie Zaza, Peter A Briss. 2005, 544-
Oxman AD, Thomson MA, Davis DA, Haynes RB: No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ. 1995, 153 (10): 1423-1431.
Sabnis SS, Pomeranz AJ, Amateau MM: The effect of education, feedback, and provider prompts on the rate of missed vaccine opportunities in a community health center. Clin Pediatr (Phila). 2003, 42 (2): 147-151. 10.1177/000992280304200208.
Lemelin J, Hogg W, Baskerville N: Evidence to action: a tailored multifaceted approach to changing family physician practice patterns and improving preventive care. CMAJ. 2001, 164 (6): 757-763.
Borgiel AE, Williams JI, Davis DA, Dunn EV, Hobbs N, Hutchison B, Wilson CR, Jensen J, O'Neil JJ, Bass MJ: Evaluating the effectiveness of 2 educational interventions in family practice. CMAJ. 1999, 161 (8): 965-970.
Assemblée nationale: Projet de loi no 90 - Loi modifiant le Code des professions et d'autres dispositions législatives dans le domaine de la santé. 2002, Éditeur officiel du Québec, 1-20.
Kalies H, Grote V, Schmitt HJ, von Kries R: Immunisation status of children in Germany: temporal trends and regional differences. Eur J Pediatr. 2006, 165 (1): 30-36. 10.1007/s00431-005-1758-0.
Estimated Vaccination Coverage with Individual Vaccines by 3 Months of Age by State and Local Area US, National Immunization Survey, 2008 (consulted on October 26, 2009). 2008, [http://www.cdc.gov/vaccines/stats-surv/nis/tables/08/tab04_3mo_iap.xls]
Massoudi MS, Walsh J, Stokley S, Rosenthal J, Stevenson J, Miljanovic B, Mann J, Dini E: Assessing immunization performance of private practitioners in Maine: impact of the assessment, feedback, incentives, and exchange strategy. Pediatrics. 1999, 103 (6 Pt 1): 1218-1223. 10.1542/peds.103.6.1218.
Boulianne N, Deceuninck G, Duval B: Pourquoi certains enfants sont incomplètement vaccinés à l'âge de 2 ans?. Rev can santé publ. 2003, 94 (3): 218-223.
Dietz VJ, Zell ER, Stevenson J: Defining delayed immunization. Pediatr Infect Dis J. 1993, 12 (4): 353-354.
Guay M, Boulianne N, Ménard S, Clouâtre A-M, Clément P, Tremblay A, Blackburn M, Lemaire J, Douville-Fradet M: Étude de validation et d'appréciation des fichiers de vaccination et de population en Estrie, en Montérégie et dans la région de Québec. 2005, Québec: Centre de recherche Hôpital Charles LeMoyne, 1-70.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/10/750/prepub
The authors would like to thank Ève Dubé, Geneviève Tremblay and Nicole Boulianne for their judicious comments on previous versions of this article. We would also like to thank Josiane Rivard for her technical support and Marie-France Richard for the layout of the manuscript. Finally, we would like to thank the company Wyeth for its financial support in the context of its Partnering for Protection program as well as the participating clinics, without whom this study could not have taken place. The funding body was not involved in any part of the study.
Competing interests for CS: research grants, honoraria and reimbursement of travel costs by the following companies: Wyeth, GlaxoSmithKline and Merck Frost. The other authors declare that they have no competing interest.
NB participated for the data collection and wrote the first draft of the manuscript. CS conceived the study, coordinated the data collection, carried out the feedback intervention, and helped in drafting the manuscript. MO and GD performed the statistical analysis and helped in drafting the manuscript. MK participated in data collection and helped in writing the first draft of the manuscript. DA, AP and CC prepared and conducted the feedback sessions for medical clinics, participated in data collection and helped in the writing of the manuscript. All authors read and approved the final manuscript.
About this article
Cite this article
Brousseau, N., Sauvageau, C., Ouakki, M. et al. Feasibility and impact of providing feedback to vaccinating medical clinics: evaluating a public health intervention. BMC Public Health 10, 750 (2010). https://doi.org/10.1186/1471-2458-10-750