Feasibility of implementation of CARD™ for school-based immunizations in Calgary, Alberta: a cluster trial
BMC Public Health volume 21, Article number: 260 (2021)
Negative experiences with school-based immunizations can contribute to vaccine hesitancy in youth and adulthood. We developed an evidence-based, multifaceted and customizable intervention to improve the immunization experience at school called the CARD™ (C-Comfort, A-Ask, R-Relax, D-Distract) system. We evaluated the feasibility of CARD™ implementation for school-based immunizations in Calgary, Canada.
In a mixed methods study, two Community Health Centres providing immunization services, including 5 schools each with grade 9 students (aged approximately 14 years), were randomized to CARD™ or control (usual care). In the CARD™ group, public health staff and students were educated about coping strategies prior to immunization clinics. Clinics were organized to reduce fear and to support student’s choices for coping strategies. Public health staff in the CARD™ group participated in a focus group discussion afterwards. We sought a recruitment rate of 80% for eligible schools, an external stakeholder focus group (e.g., school staff) with 6 or more individuals, 85% of individual injection-related data acquisition (student and immunizer surveys), and 80% absolute agreement between raters for a subset of data that were double-coded. Across focus groups, we examined perceptions of acceptability, appropriateness, feasibility and fidelity of CARD™.
Nine (90%) of eligible schools participated. Of 219 students immunized, injection-related student and immunizer data forms were acquired for 195 (89.0%) and 196 (89.5%), respectively. Reliability of data collection was high. Fifteen public health and 5 school staff participated in separate focus groups. Overall, attitudes towards CARD™ were positive and compliance with individual components of CARD™ was high. Public health staff expressed skepticism regarding the value of student participation in the CARD™ system. Suggestions were made regarding processes to refine implementation.
While most outcome criteria were satisfied and overall perceptions of implementation outcomes were positive, some important challenges and opportunities were identified. Feedback is being used to inform a large cluster trial that will evaluate the impact of CARD™ during school-based immunizations.
The trial is registered at ClinicalTrials.gov (NCT03948633); Submitted April 24, 2019.
School-based immunizations are an efficient and accepted method of immunizing youth in many countries worldwide, including Canada . Until recently, the immunization experience of students in this context was not addressed as a vaccine hesitancy concern. We now know that students experience fear, pain and other anxiety-related symptoms during vaccine injections and that having such negative experiences can undermine immunization acceptance and compliance . To address this issue, we developed a person-centred framework  which promotes student participation and coping during immunization called the CARD™ (C-Comfort, A-Ask, R-Relax, D-Distract) System . CARD™ incorporates evidence-based interventions and student preferences in the planning and delivery of immunizations in the school setting. Students are educated about CARD™ ahead of immunization day and then asked what CARDs they want to play on immunization day. They select from the four different letter categories - for instance, they may select the ‘Distract’ card and use their cell phone as a distraction agent.
In a controlled cluster trial conducted in Niagara, a small urban region of Ontario, Canada, CARD™ reduced high levels of fear and dizziness in grade 7 students and was so valued by the students and public health nurses that all schools in that region are now using this program [5, 6]. In the current project, we explored how to customize CARD™ before scaling the implementation. This is important given the considerable differences that may exist in: processes used to deliver immunizations among geographical regions, developmental and maturity levels of the children across grade spans targeted for immunizations at school, as well as socioeconomic and cultural factors. This study addresses tailoring CARD™ to a large urban region, Calgary, Alberta, whereby all routine vaccines are administered across the province by one health provider, Alberta Health Services (AHS), either in their public health offices or at school. This contrasts with Niagara, whereby only grade 7 immunizations are typically administered by the public health authority while earlier childhood immunizations are given in physicians’ offices. In addition, Alberta targets both grade 6 and grade 9 students for mass school immunizations and includes a diverse population, in terms of socioeconomic status and cultural background. This work was the second of a three-phase project aimed at customizing and implementing CARD™ for grade 6 and 9 school-based immunizations in Calgary. In phase one, we sought input from stakeholders to inform alterations to CARD™ resources and implementation approaches for school-based immunizations in grade 6 and in grade 9 students [7, 8]. In the current project (phase two), we trialed CARD™ implementation in grade 9 students in a small number of schools using a randomized controlled trial design. Our specific objectives were to test recruitment, data collection/study processes and implementation outcomes (acceptability, appropriateness, feasibility, fidelity). The results will inform a large randomized cluster trial evaluating the impact of CARD™, which is the final phase of the project (phase three).
The Consolidated Framework for Implementation Research (CFIR)  was used to guide items to probe in data collection and analysis related to implementation barriers and facilitators. CFIR is a comprehensive framework that identifies 31 constructs that positively and negatively influence implementation of complex interventions. We applied the constructs identified in CFIR to the taxonomy of implementation outcomes proposed by Proctor to describe the overall implementation success of CARD™ . Four of the 8 categories of implementation success were included as intermediate outcomes of CARD™‘s treatment effectiveness: 1) acceptability: satisfaction with various aspects of the intervention (e.g., content, complexity, comfort, delivery, credibility); 2) appropriateness: perceived fit of intervention; 3) feasibility: extent to which CARD™ can be carried out; and 4) fidelity: compliance and quality of implementation. Our expectation is that the effectiveness of CARD™ will be influenced by the success of implementation. Hence, it is important to address identified implementation barriers and challenges because they provide the context for interpreting clinical effectiveness.
Design, setting and participants
We employed quantitative and qualitative methods, including a small feasibility cluster trial and focus group interviews. AHS services are segregated into five health zones where Community Health Centers (CHCs) or service locations offer a variety of programs at the community level, including school immunization programs. The study setting included two CHCs in Calgary providing immunization services to jurisdictional schools. These two CHCs were selected from a pool of 8 primarily urban CHCs randomized to intervention (CARD™) or control (usual care) groups for phase three of the project (large cluster trial). The present study took place during the third round of grade 9 immunizations in the spring of 2019. Within both CHCs, 5 schools were randomly selected by drawing folded slips of paper with the names of schools written on them from an opaque envelope including all available schools. The principals of selected schools were subsequently invited to participate via an invitation letter.
In the Calgary school immunization program, public health school nurses are responsible for liaising with schools to organize immunization clinics, deliver education, and oversee the clinics. Immunizing nurses administer the vaccines. Managers oversee staffing, training and work processes. Administrative staff support the program, including organizing supplies and attending the clinics and assisting with workflow and supervising students.
Ethical approval was obtained from the University of Toronto Health Sciences Research Ethics Board (REB) and the REBs for the School Boards of participating schools. Informed consent was obtained from all focus group participants. Consent was waived by the REB for the remainder of the study data to allow collection of population level data. Hence, parent and/or student consent was not required. Principals, however, were able to refuse participation of their school.
Table 1 provides a summary of the key phases and activities undertaken by public health staff in the intervention CHC (also reviewed in staff training video ). Briefly, staff involved in the immunization program were trained in the intervention in a 1-day workshop, which included presentations by study team members, including content experts and formally appointed internal implementation leaders, about: rationale for the project, scientific evidence, alignment with organizational values/mission, relevant policies and work processes. They also reviewed point of care resources (e.g., pamphlets, videos). Draft work process checklists to be used to organize and track immunization and CARD™-specific activities were incorporated into the training to allow for practice and problem-solving of potential barriers. During the training, staff provided comments which were used to inform the final implementation plan and process checklists used in the present study. All final resources were provided to staff via immediate managers, with support provided by two formally appointed local implementation leaders (TS, CL).
CARD™ was then integrated into school immunization program activities for included schools, including pre-immunization preparation (e.g., planning of clinic spaces, student education). Figure 1 displays a resource distributed to students by public health nurses during student education sessions prior to the immunization clinic. Students used the pamphlet to record preferred coping strategies for the upcoming clinic. On immunization day, staff set-up clinics, organized student flow and modified interactions with students according to the principles of CARD™ (Table 1). CARD™-trained staff did not communicate about the intervention with those in the control group, nor did they deliver care to schools in the control group.
Control group (usual care)
Public health staff in the control CHC were unaware of the intervention and followed usual practices, which excluded systematic approaches to: planning immunization clinic spaces, education of students about CARD,™ reducing cues of fear on immunization day (e.g., clinic-setup), and active student participation during immunization injections.
All tools and methods were adapted from our prior research with CARD™ . Group-specific (CARD™ and control) versions of process checklists were completed by public health staff to guide and document their activities, including compliance with core components of CARD™. This included a student education checklist with details about CARD™ education provided to students in CARD™ schools ahead of time, and an immunization day checklist with details about what was done on immunization clinic day. During student immunizations, all nurses additionally completed a nurse feedback form after each interaction with a student, whereby details regarding individual student gender, vaccine(s) administered, and coping strategies used by students during injections were documented. Separately, all students independently completed an immunization symptom survey reporting on their level of fear, pain, and dizziness, each rated on separate 0–10 numerical rating scales. Data obtained from these checklists and forms are displayed in Tables 2, 3, 4 and 5 (see Results section of manuscript). Blank checklists and forms are available on request.
An observer independently recorded the same data as public health staff for selected process checklists and data forms (described further in Outcomes section of manuscript); this was planned for at least 2 schools (up to 4 schools) and for approximately 5–10% of students undergoing immunization, including intervention and control schools.
After clinics were undertaken in study schools, a purposive sample of public health staff implementers participated in a focus group at a public health head office. Individuals from different roles were sought to allow for a variety of perspectives, including school nurses, immunizing nurses, managers and administrative staff. The focus group discussion was facilitated by one of the study team members (AT) using a semi-structured interview guide (available upon request) and focused on CARD™ implementation, including components of CARD™ that were implemented and how they worked, confidence in ability to implement CARD™, CARD™ processes, and required resources. Public health staff rated their agreement with attitude and belief statements about CARD™ using a 5-point Likert scale (anchors: 1 = strongly disagree, 5 = strongly agree). Suggestions for phase three of the project were also recorded. Invitations for other stakeholder groups (i.e., school staff, parents, students) to participate in separate focus groups at the school were overseen by local implementation leaders, with input of the school nurses. Focus group discussions with external stakeholders and discussion notes from study team meetings were used to supplement the data and identify additional suggestions for refinement of CARD™ implementation.
Below is a summary of the outcomes, level of analysis (cluster-school or individual), and associated a-priori success benchmarks.
Recruitment and attrition rates
Percent of schools recruited (via principal agreement), cluster level: 80% or more
Focus group recruitment (either school staff, parents, or grade 9 students), individual level: 6 individuals or more
Percent of student immunization symptom surveys returned with data (yes/no), individual level: 85% or more (primary outcome)
Percent of nurse feedback forms returned with data (yes/no), individual level: 85% or more
Percent of core components of CARD™ delivered (yes/no), cluster level: 80% or more
Perceptions about CARD™, assessed via agreement with attitude statements about CARD™ (5-point Likert scale) and focus group discussion, individual level. Information was supplemented with focus groups in other stakeholder groups, as available. Suggestions for future refinement of implementation and reporting were tracked from focus group feedback and discussion notes from weekly team meetings to inform phase three of the project.
Reliability of data collection
Percent agreement with 17 items included in the CARD™ compliance checklist (yes/no), cluster level: 80% or more. Data were abstracted from two process checklists containing the relevant information: student education process checklist and immunization day checklist.
Percent agreement with 18 items included in the nurse feedback form (yes/no), individual level: 80% or more. These items track student gender, vaccination administration details, and coping strategies used.
The a priori sample size calculations were based on the primary outcome (i.e., percent of student immunization symptom surveys completed). A sample size of 215 (108/group) was required to obtain a value of 85% with a 2-sided 95% Confidence Interval (CI) of 80 to 90%. Clustering was not taken into account. Ten schools (5/group) were invited with the expectation that the requisite number of students would be included, with a buffer to include more students to account for fluctuation in grade 9 census (approximately 260 altogether). Focus group interviews were planned for 3–24 public health staff and managers, and 3–12 individuals in all other stakeholder groups.
Quantitative analysis consisted of descriptive statistics (central tendency and variation). Qualitative analysis followed a stepwise process that included the use of two theoretical frameworks: CFIR  and Proctor’s implementation outcomes framework . First, notes, process checklists and verbatim transcripts of focus group discussions were reviewed to obtain an overall understanding of all of the data. Then focus group transcripts were reviewed by three researchers (AT, CL, VS) and coded using directed content analysis with CFIR as the coding framework. NVivo 10 was used to organize the results. Next, the CFIR codes were reviewed by three team members (AT, CL, CMM) and subsumed within the four pre-specified overarching implementation domains (or themes) selected a-priori to describe overall implementation success: acceptability, appropriateness, feasibility and fidelity. One researcher (AT) then reviewed all of the data again and made slight modifications to the coding scheme, which were subsequently reviewed by other researchers (CMM, CL, VS) until a consensus was reached. The results of the qualitative analysis are organized by implementation outcome (i.e., acceptability, appropriateness, feasibility and fidelity), along with the relevant implementation barriers or facilitators (as defined in CFIR) identified within each of the outcomes.
Figure 2 displays the Flow Diagram for participants in the study. The data were collected between April 25 and June 11, 2019. Altogether, 9 schools participated: 4 were in the CARD™ group and 5 were in the control group. The geographical zones of included schools revealed a similar mean number of grade 9 students in each school (103 and 110) and percentage of immigrants (33 and 33.6%) in the groups. The median household income was $115,000 Canadian dollars (CAD) and $109,000 CAD, respectively. Of 219 students recorded as immunized, 90 were in the CARD™ schools and 129 were in the control schools (Fig. 2).
Table 2 displays the characteristics of immunized students, as obtained from student immunization symptom surveys and nurse feedback forms on immunization day. Two separate focus groups were conducted: one with 15 public health staff members in the intervention CHC and one with 5 school staff members from one intervention school. Table 3 displays the characteristics of focus group participants.
Recruitment and attrition
Of 10 school principals approached for participation, 9 (90%) agreed. In the one school that refused participation, the principal was concerned about additional workload for staff imposed by CARD™. There were 195/219 (89.0%) individual student immunization symptom surveys and 196/219 (89.5%) nurse feedback forms returned with partial or complete data. There were some issues, however, verifying the number of students immunized (records of the number of eligible students did not match number of data forms retrieved in Control schools). This may account for the imbalance between groups in incomplete data (Fig. 2). In the only external stakeholder focus group carried out, an insufficient number of individuals (less than 6) participated. No other stakeholder groups were sought for participation due to time constraints.
Public health staff self-reported compliance with individual components of CARD™ was calculated to be 91.2% (Table 4). Attitude survey results revealed positive perceptions about acceptability, appropriateness, feasibility and fidelity of CARD™ implementation, as evidenced by mean Likert scores of greater than 4 (agree) out of 5 (strongly agree) for most questions (Table 5). However, scores were lower for perceptions of effectiveness of CARD™ (mean, 3.8), whether it was realistic to continue to use CARD™ (mean, 3.5), and CARD™-associated documentation workload (mean, 3.4).
Qualitative feedback by public health staff and school staff is summarized below, organized by each implementation outcome domain and the specific CFIR barriers/facilitators identified within them. Selected quotations are provided in Table 6; participants are identified by group (PH = public health staff, SS = school staff) and number (01, 02, 03, etc).
Four CFIR constructs were identified related to positive and negative perceptions of acceptability of CARD™; 1) relative advantage compared to usual care; 2) design quality and packaging and delivery (i.e., how the intervention is bundled, presented and assembled); 3) individual stage of change (i.e., phase of progression in the use of the intervention); and 4) identification and priority of patient needs and resources. Nurses and school staff reported that many components of CARD™ were advantageous when compared to usual practices, including: having fewer waiting students, use of a separate waiting area for students with chairs, performing team introductions to classes prior to clinic commencement, enhancing privacy in the clinic, providing school announcements reminding students of immunization day, assessing baseline levels of fear in students, and reducing the potential for discipline issues resulting from negative immunization experiences.
Nurses perceived that student and parent education may have been inadequate and believed this was due to the materials used and/or their delivery method, including timing relative to immunizations. In addition, nurses did not consistently attribute positive immunization experiences with student self-selected coping choices. For example, an immunizing nurse reported that one student was more nervous with CARD™. One nurse commented on increasing her own buy-in over time that was influenced by re-review of CARD™ educational tools and positive feedback received from school staff. School staff reported the benefits of educating students ahead of time and the empowerment of students in the immunization process.
Three CFIR constructs related to perceptions of appropriateness: 1) external policy and incentives that spread the intervention; 2) reflecting and evaluating on the process of implementation; and 3) compatibility or degree of fit. Nurses reported that CARD™ was aligned with current activities undertaken by school nurses and that it helped to strengthen relationships with schools. Encouraging involvement of immunizing nurses in regular CHC staff meetings was recommended for building and strengthening relationships among staff. It was suggested that student education might be more effective in the fall before they experience grade 9 school immunization clinics.
Hesitancy was expressed by immunizing nurses regarding a change to practice announced by a senior manager to inject 2 different vaccines in the same arm (rather than separate arms), despite rationale for safety and efficiency provided by the manager. School administrators reported a willingness to work with public health staff to plan and accommodate requests.
One participant identified a new point-of-care immunization data entry program being implemented concurrently that increased the amount of time required for each immunization encounter, which was counter to some CARD™ measures intended to reduce time.
Six CFIR constructs related to perceptions of feasibility: 1) compatibility; 2) design quality and packaging; 3) complexity of the intervention; 4) adaptability of the intervention; 5) available resources to implement the intervention; and 6) reflecting and evaluating on the process of implementation. With respect to compatibility, immunizing nurses reported that the interaction between immunizing nurses and students fit into usual workflows. One vice principal, however, mentioned sitting with unsupervised students waiting to be immunized to assist with anxiety. Several nurses mentioned that items included in the distraction kit (squishy ball, fidget spinner, word puzzle) were inappropriate for grade 9 students. Nurses and managers commented on difficulty implementing some interventions related to school factors, including physical layout and behavior of students and staff. Nurses reported making adaptations to prescribed implementation processes with the intention of trying to achieve the underlying goal, such as; introducing themselves to students after taking them out of class rather than before and attaching completed student CARD™ pamphlets to immunization consent forms to prompt nurses about student choices. Nurses and managers identified regular staff meetings as a good venue to debrief about CARD™ implementation to inform reflections and additional refinements. One of the managers stated that additional resources were required to implement CARD™. Nurses reflected on their experiences with specific implementation processes and offered ways to improve them.
One CFIR construct related to intervention fidelity: 1) executing (carrying out or accomplishing the implementation according to plan). Nurses reported variations in compliance and quality of implementation of different aspects of CARD™. For instance, one nurse described learning to educate students more effectively after observing that students filled out their CARD™ pamphlet as if it was ‘a test’ rather than choosing their own coping strategies. Sometimes, interventions were not executed, and nurses suggested that poor fidelity with implementation may have impacted effectiveness.
Nurses reported that some students did not appear to be interested in CARD™ education and that students did not consistently choose to play any cards during immunization. Nurses also reported using usual coaching approaches during immunization injections whereby they directed students rather than engaging them in making their own choices as recommended by CARD™.
Suggestions for future cluster trial
Suggestions regarding additional refinement for the cluster trial are summarized in Table 7. Briefly, they include activities overseen by implementation leaders to foster a positive implementation climate, booster training for staff, additional ‘how to’ resources for operationalizing different components of CARD™ and data collection procedures, and minor adjustments to the student education.
Reliability of data collection
Overall percent agreement for compliance among 17 of 20 individual components of CARD™ included in the study was 100%. Altogether, percent agreement for 15 nurse feedback forms (6.8% of all immunized students) was 92.6% for the 18 targeted variables on the form. Discrepancies above > 20% were noted for certain coping strategies, including verbal distraction, external distraction and deep breathing. It was noted by observers at one school that some non-fearful students were not offered cards to play during injections.
We evaluated study procedures and implementation outcomes for a novel evidence-based framework for delivering vaccines at school in preparation for a future cluster trial. This CARD™ system framework integrates specific activities into usual school immunization planning and administration procedures that together promote person-centred care and coping. Overall, school recruitment was high as were retrieval of both student and nurse-reported outcome data. However, one principal in an intervention school declined participation due to concerns about workload. In addition, there was some confusion regarding the total number of immunized students because of a lack of accounting of all data forms. Self-reported compliance with the main components of CARD™ and reliability of data collection were high. There were some parts of the nurse feedback form, however, related to specific coping interventions, that were not consistently recorded between immunizing nurses and observers. Overall, feedback from participants was positive regarding perceptions of acceptability, appropriateness, feasibility and fidelity. However, there was skepticism regarding the value of components of CARD™ that involved students, including education about CARD™ ahead of time and student-led coping strategies during immunization injections. In addition, nurses reported challenges carrying out some aspects of CARD™ and anticipated future challenges, some of which related to limits imposed by school physical layout and routines. Limited feedback obtained from school staff revealed support for CARD™ and willingness to collaborate with public health staff in its implementation. Suggestions were made to refine future implementation and study processes.
These results are consistent with prior research demonstrating that CARD™-informed alterations in the school immunization delivery process are possible and can have positive effects on public health staff perceptions of clinic success [5, 6]. In particular, staff reported satisfaction with changes in the physical set-up and flow of students to reduce cues of fear. In addition, CARD™ was perceived as promoting interactions and collaboration between individuals in different roles within public health (school nurses and immunizing nurses) as well as across organizations (public health staff and school staff). These interactions were viewed as feasible within current work processes. Importantly, school staff were supportive of CARD™ and willing to work with public health staff to plan immunization clinics that followed the principles of CARD™.
With respect to implementation fidelity, some components of CARD™ were inconsistently applied such as education of students and offering non-fearful students cards to play during vaccine injections. Staff also reported some logistical and physical challenges related to workflow and school factors (e.g., physical layout and routines). This led to adaptations to locally prescribed implementation approaches or omitting specific activities altogether. CARD™ encompasses numerous process components, many of which were new to staff; therefore, staff may not have grasped if and how the locally prescribed implementation approaches could be adapted and to what extent. Additional training and support are recommended to provide staff with the rationale for inclusion of all components of CARD™ and options for how to implement them that consider the likelihood of achieving desired outcomes. Including staff in planning and trialing new approaches, particularly for situations whereby the optimal course of action is unclear, may also increase engagement, collaboration, evaluation and self-efficacy. This, in turn, can further enhance implementation success.
In general, nurses were skeptical about including students as active partners in the process. They reported that some students did not receive, understand or appreciate CARD™ education, and did not come prepared to play their cards during the immunization clinic. At the same time, when students did play their cards, nurses expressed uneasiness about having students leading their own coping strategies, particularly if they conflicted with the approaches that the nurses or parents preferred. This may have contributed to less positive attitudes regarding the effectiveness of CARD™ and how realistic it was to continue to use it. Engaging students as active partners during immunization and coaching them on their preferred coping strategies may represent a significant change in perspective for many nurses who are accustomed to leading interactions with students and directing them to use nurse-selected coping strategies. This finding contrasts with the experience in Niagara, whereby student involvement was perceived as a major benefit by both nurses and students [5, 6]. This finding represents an opportunity for further evaluation of staff perceptions of what is intended by a person-centred approach to immunization delivery as well as student perceptions of being included . Importantly, students in the present study, unlike in Niagara, would have had prior experiences with immunizations at school, including in grade 9, which may have shaped how they reacted to the nurses and the new program during the education and then on immunization day.
In addition, nurses seemed uncertain about the role of CARD™ in fearful vs. non-fearful students. Specifically, it was unclear to them whether students who were not fearful should play their cards. A significant part of CARD™ addresses student coping with injection-related fear and pain; however, it is a more comprehensive framework that incorporates students as active participants in the immunization process, addressing their concerns and giving them choices and control over how immunizations are delivered in order to meet their needs and preferences. There are no “cut off” fear scores as this is a universal intervention designed to be relevant for all students and is consistent with currently espoused models of health care delivery that stress participation of individuals. Given the limited number of interactions nurses had with students who self-selected coping strategies and the late introduction of CARD™ relative to school immunization clinics in the current academic year, nurses may feel more comfortable with student-led participation with more experience and if the timing of CARD™ implementation coincides with the first immunization clinic in the academic year. To assist with this transition, additional resources and support reinforcing the rationale and assisting with practice are recommended to facilitate learning and lead to more positive attitudes about this approach. Our prior work in grade 9 students leading up to this study suggested acceptability of CARD™ for this age group . As grade 9 students in the present study were not consulted about their perceptions of CARD™, additional research is recommended to find out more about how to customize CARD™ to meet their needs and preferences.
Reliability testing revealed some discrepancies in the frequency of reporting of specific coping strategies between raters, suggesting inconsistent operational definitions were being used when recording the data. With respect to verbal and external distraction, it was determined that the timeframe for the use of the strategy largely contributed to differences in ratings (i.e., whether undertaken before or during the injection). With respect to deep breathing, two separate interventions were identified that use the same name: 1) one deep breath taken just before injection, and 2) several deep belly breaths taken before and during injection. Based on these findings, there is potential for random error in this variable in the cluster trial. There is also potential for bias because of systematic differences between groups in staff training about coping strategies. Those in the CARD™ group receive more training and this may lead to systematic differences in how they recorded the use of various coping strategies. As there was no reliability data collected in control schools, the potential for bias could not be explored. The pattern of use of coping interventions, while not formally analyzed between groups in the present study, may be suggestive of bias in the reporting of some strategies (see Table 2). Potential biases may lead to a divergence in the results between groups in utilization patterns for different coping interventions and student symptoms. Ways to mitigate both error and bias are highly recommended for the cluster trial.
This study included a limited number of public health staff implementers, schools and students and not all parts of CARD™ were included. In addition, few students requested any coping interventions. This may have prevented us from identifying all potential challenges with implementation processes in preparation for phase three of the project. Future challenges may include: 1) ensuring staff educate all students due to the vast number of schools and students; 2) fidelity with carrying out certain activities because of perceived need, resource or logistical challenges; and 3) role of school staff in implementation success, including cooperation and collaboration with requested activities. Frequent communication between implementing staff and local study leaders and regular study communication (newsletter, Q&A) are recommended to identify and address arising implementation issues.
There are numerous strengths of the study related to methodologic rigor, including: randomization of schools, blinding of key individuals, inclusion of all immunization clinic-related data (i.e., all immunized students), pragmatic implementation, use of theoretical frameworks to guide data collection and interpretation, and inclusion of multiple data sources. In addition, the research team included health care providers from across disciplines and roles with experience with school immunizations. Together, these factors contributed to a more comprehensive inquiry into the implementation context for CARD™ in Calgary. A number of recommendations were made that relate to staff training and study procedures for future implementation efforts. With respect to training, additional ‘how to’ tips were recommended that identify different ways to operationalize core components of CARD™ with careful consideration of compatibility with practice. This might also include information about the associated likelihood of achieving intended outcomes with each of the options. With respect to data collection, procedures that allow for accounting of all data forms were recommended. A checklist for managers was developed to assist with overseeing of training and study procedures.
This study tested recruitment procedures, data collection and study processes and implementation outcomes (acceptability, appropriateness, feasibility, fidelity) for a future cluster trial evaluating a new framework for vaccine delivery at school called the CARD™ system. Overall, study processes were successful, and participants had positive perceptions about CARD™. The findings support the ability to carry out a large cluster trial that will evaluate the effectiveness, fidelity and cost of CARD™ implementation in Calgary. Improving the experience of students undergoing immunization has the potential to reduce vaccine hesitancy, which will, in turn, improve vaccine uptake. This work is particularly relevant now, as surveys suggest that vaccine hesitancy has been steadily increasing during the pandemic, and may threaten the success of immunization programs. Additional research is therefore also recommended to adapt CARD™ for adult COVID-19 immunization.
Availability of data and materials
The datasets generated for this manuscript are not publicly available. Permission was obtained by the overseeing ethics board, Alberta Health Services and participating schools to collect the data; however, permission was not granted to make the datasets public.
Alberta Health Services
Consolidated Framework for Implementation Research
Community Health Centre
Vandelaer J, Olaniran M. Using a school-based approach to deliver immunization—global update. Vaccine. 2015;33:719–25.
McMurtry CM, Pillai Riddell R, Taddio a RN, Asmundson GJG, noel M, et al. far from “just a poke”: common painful needle procedures and the development of needle fear. Clin J Pain. 2015;31(10 Suppl):S3–11.
Sharma T, Bamford M, Dodman D. Person-centred care: an overview of reviews. Contemp Nurse. 2015;51:107–20.
Taddio A, McMurtry CM, Bucci LM, MacDonald N, Ilersich ANT, Ilersich ALT, et al. Overview of a Knowledge Translation (KT) project to improve the vaccination experience at school: The CARD™ System. Paediatr Child Health. 2019;24(Suppl 1):S3–18.
Freedman T, Taddio A, Alderman L, McDowall T, deVlaming-Kot C, McMurtry CM, et al. The CARD™ System for improving the vaccination experience at school: Results of a small-scale implementation project on student symptoms. Paediatr Child Health. 2019;24(Suppl 1):S42–53.
Taddio A, Alderman L, Freedman T, McDowall T, McMurtry CM, MacDonald N, et al. The CARD™ System for improving the vaccination experience at school: Results of a small-scale implementation project on program delivery. Paediatr Child Health. 2019;24(Suppl 1):S54–67.
Logeman C, Taddio A, McMurtry CM, Bucci L, MacDonald N, Chalmers G, et al. Student feedback to tailor the CARD™ system for improving the immunization experience at school. Children. 2020;7(9):126.
Logeman C, Taddio A, Bucci L, Calgary CARD™ Implementation Team. Public health staff perspectives on the CARD™ system to improve the vaccination experience at school: a qualitative study. Public Health 2020; October 14–16, 2020. (virtual meeting due to COVID-19).
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health. 2011;38:65–76.
School Vaccinations – Improving the vaccination experience at school. https://www.youtube.com/watch?v=FXj6ELi4BVg&feature=youtube. Accessed: 16 April 2020.
The authors wish to thank all of the individuals who participated in this project.
The study was funded by a Canadian Institutes of Health Research Foundation Grant awarded to A Taddio (FRN 159905). The funding agency had no input into the study.
Ethics approval and consent to participate
Ethical approval was obtained from the University of Toronto Health Sciences Research Ethics Board (REB) and the REBs for the School Boards of participating schools. Written informed consent was obtained from all focus group participants. Consent was waived by the REB for the remainder of the data to allow collection of population level data. Hence, parent and/or student consent was not required. Principals, however, were able to refuse participation of their school.
Consent for publication
AT reports a Section 9 Trademark No. 924835 for CARD™. LB reports that Immunize Canada received grants from Pfizer Inc., Merck Canada, Sanofi Pasteur, Seqiris, and Glaxo-Smith-Kline unrelated to this work. The other authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Taddio, A., Coldham, J., Logeman, C. et al. Feasibility of implementation of CARD™ for school-based immunizations in Calgary, Alberta: a cluster trial. BMC Public Health 21, 260 (2021). https://doi.org/10.1186/s12889-021-10247-4
- School immunization/vaccination
- Pain management
- Vaccine hesitancy
- Feasibility study