Development and evaluation of a leadership training program for public health emergency response: results from a Chinese study
© Wang et al; licensee BioMed Central Ltd. 2008
Received: 16 October 2007
Accepted: 30 October 2008
Published: 30 October 2008
Since the 9/11 attack and severe acute respiratory syndrome (SARS), the development of qualified and able public health leaders has become a new urgency in building the infrastructure needed to address public health emergencies. Although previous studies have reported that the training of individual leaders is an important approach, the systemic and scientific training model need further improvement and development. The purpose of this study was to develop, deliver, and evaluate a participatory leadership training program for emergency response.
Forty-one public health leaders (N = 41) from five provinces completed the entire emergency preparedness training program in China. The program was evaluated by anonymous questionnaires and semi-structured interviews held prior to training, immediately post-training and 12-month after training (Follow-up).
The emergency preparedness training resulted in positive shifts in knowledge, self-assessment of skills for public health leaders. More than ninety-five percent of participants reported that the training model was scientific and feasible. Moreover, the response of participants in the program to the avian influenza outbreak, as well as the planned evaluations for this leadership training program, further demonstrated both the successful approaches and methods and the positive impact of this integrated leadership training initiative.
The emergency preparedness training program met its aims and objectives satisfactorily, and improved the emergency capability of public health leaders. This suggests that the leadership training model was effective and feasible in improving the emergency preparedness capability.
Since the 9/11 attack, mad cow disease, severe acute respiratory syndrome (SARS), and avian influenza outbreaks, public health emergencies have become an utmost threat to communities worldwide. Moreover, because public health provides population-focused services to entire communities rather than individualized care [1, 2], an increasing need exists for public health personnel capable of leading efforts to ensure the effectiveness and quality of these services . During the past decade, there has been a growing interest in improving the emergency preparedness capability of public health leaders in the United States and other countries [4–10]. Although the Chinese government carried out a series of emergency preparedness education and training programs to improve the capability of public health leaders after the SARS outbreak, it remains unclear if these training programs are feasible and effective in improving the capability of emergency preparedness.
Previous studies showed that the emergency preparedness capability of public health leaders was insufficient in China [11, 12]. In order to change the current situation and to improve emergency preparedness capability of public health leaders, one emergency preparedness training program for public health leaders was developed and supported by the Ministry of Health (MOH) of the People's Republic of China and the World Health Organization (WHO). The study was carried out by Tongji Medical College Emergency Institute (TMCEI) from 2006 to 2007. The training was completed in 2006, and the follow-up survey was conducted 12 months later. The purpose of this article was to evaluate the effectiveness of a leadership training program in improving the emergency capability, and then to develop a participatory leadership training model for public health emergency response.
Methods and Subjects
Aims and objectives
The aims and objectives of leadership training were carefully designed in consultation with the educational and training experts who have profound knowledge of the public health emergency response plan and the training program. The overall goals of the project were to: (1) develop and deliver a participatory leadership training program on responding to public health emergencies, and (2) test whether this pilot leadership training program prepares public health leaders to better respond to emergencies. The objectives of the training were subjected to continuous monitoring and evaluation during the training period.
Participants and trainers
The research was approved by Tongji Medical College Ethics Committee, and all participants in the study agreed with the relevant training data for the study. Forty-three public health leaders from Health department or Centers for Disease Control and Prevention (CDC) in Hubei, Henan, Hunan, Jiangxi, and Anhui, participated in the leadership training program in 2006. There were 20 participants in the class held on July, 23 and in the class held in August. Each class lasted 14 days (112 hours). Two participants did not complete their training for reasons unrelated to the training, and they were not included in the evaluation (N = 41, 95.35% response rate). Trainers came from MOH, WHO, Chinese CDC, Fudan University, Wuhan University, and Huazhong University of Science and Technology. The selection of trainers was based on their expertise in the field of public health emergency response, related training programs and their involvement in continuous consultations on health service programs, both educational and promotional.
Course and content
Learning objectives: emergency preparedness training program for public health leader
Core competencies for public health leader
Describe the public heath role in emergency response in a range of emergencies that might arise (e.g., This department provides surveillance, investigation, and public information in disease outbreaks and collaborates with other agencies in biological, environmental in weather emergencies).
Recognize unusual events that might indicate an emergency and describe appropriate action (e.g., communicate clearly within the chain of command.)
Identify limits to own knowledge/skill/authority and identify key system resources for referring matters that exceed these limits.
Describe his/her functional role(s) in emergency response and demonstrate his/her role(s) in regular drills.
Identify and locate the agency emergency response plan (or the pertinent portion of the plan).
Explain the interaction of central and local agencies and describe communication role(s) in emergency response (media, within agency, general public and personal)
Evaluate every emergency response drill/emergency response to identify needed internal/external improvements.
Describe the chain of command and management system ("incident command system" or "similar protocol") for emergency response in the jurisdiction.
Apply creative problem solving and flexible thinking to unusual challenges within his/her functional responsibilities and evaluate effectiveness of all actions taken
Format and process
Various training methods were used, including case studies, workshops, tutorials, seminars, group discussions, role-playing, drilling, and fieldwork. Formal lecturing was the method least used. The training center was equipped with modern audiovisual aids designed for training purpose. The training logistics and general services, such as transportation and housing, were provided free of charge to the participants.
The purpose of evaluating any program is to identify its strengths and weaknesses so that modifications can be made. This was especially true for this program which was designed to be innovative, relevant, flexible and not just a "one size fits all". The program was evaluated by two different methods: anonymous questionnaires and semi-structured interviews.
Through an anonymous questionnaire, the following outcomes were measured and investigated: individual basic information, knowledge levels, and self-assessment of skills regarding emergency preparedness. The questionnaire was designed by experts in the field of training programs and continuous consultation on emergency management. In order to assess the questionnaire, a pilot test was carried out among other public health leaders who did not participate in the training, and based on their feedback, subsequent modifications were made. A total of forty questions assessed participant's knowledge on the public health emergency competencies according to the "Core Public Health Worker Competencies for Emergency Preparedness and Response"  and "Ten Essential Public Health Services" , which consisted of basic public health science and culture knowledge, analytic/assessment knowledge, program planning and management knowledge. For a correct answer to a question, the participant received one point, whereas an incorrect answer received no points. Additionally, respondents were asked fifteen self-assessment questions so that the frequency of their use of each of the competencies could be measured . Responses were rated on an ordinal scale (1 = "very low", 2 = "low", 3 = "Average", 4 = "high", 5 = "very high"). Participants completed the first measurement (pre-test, baseline) on the first day of training. The post-training measurement (post-test) was conducted at the end of the last day of training. For the third measurement (Follow-up test), the participants were mailed a copy of the survey with a self-addressed return envelope 12-months after completion of the program.
The training program was also subjected to continuous monitoring and evaluation through semi-structured interviews. The inclusion of the trainees in the evaluation process was extremely helpful in updating and modifying the program, for both the betterment of the program and the satisfaction of all the participants. The items addressed in the semi-structured interviews were as follows: (1) the scientific methods offered, (2) the technical material presented, (3) the performance of the trainers, (4) the benefits derived by the participant, (5) the use of the audiovisual aids, (6) the strengths and weaknesses of the session, (7) final critical comments and remarks, and (8) the training model. The forms were distributed at the end of each session to be completed anonymously by each participant. The forms were immediately analyzed and the results shown to the trainer who had conducted the session. If any deficiencies were revealed, the necessary modifications were made immediately. Evaluation of workshops and fieldwork was carried out in a similar fashion. Feedback of the results of the evaluation was given to the participants.
The responses to the forty questions regarding emergency knowledge were reported as scores. The scores of self-assessment were derived for each domain by participants' response to all the frequency questions separately. Repeated-measures analysis of variance was used to test differences between pre-test, post-test, and follow-up test. The data from the semi-structured interviews was categorized and analyzed by three authors independently using the triangulation method, and their individual results were compared and discussed, and a consensus was reached. Data was analyzed by one way ANOVA with SPSS12.0 (SPSS. Inc., USA) for Windows. P-value of < 0.05 was used as the significance level.
Forty-one of the study participants completed the entire training program and represented public health leaders from Hubei, Henan, Hunan, Jiangxi, and Anhui. Most respondents were male (N = 36, 87.80%) and over half (N = 34, 82.93%) had earned a bachelor's degree or master's, of which one-fifth possessed MPH degrees. Additionally, most participants (N = 33, 80.49%) had more than five years of experience as public health leaders. Some (N = 27, 65.85%) had participated in interrelated training, but the previous trainings had been conducted about 12–24 months prior to the study. The results of reliability assessment have shown that test-retest reliability and the internal consistency of questionnaires was accredited to some extent (Test-retest reliability of pre-training = 0.87, Cronbach's alpha > 0.63, respectively). The results of related analysis indicated that the construct validity of the questionnaires is of high quality (Related coefficient fluctuated between 0.37 and 0.79, P < 0.05) [21, 22].
Change in self-assessment of skills among the study respondents
Self-assessment of skills
skills Level, Mean(SD)
Cope with and lead changes in public health practice
Match the skills and knowledge of public health workers with appropriate tasks
Deal with cultural and ethnic diversity in the context of access to health services
Mobilize resources in the community needed to increase access to public health services
Communicate clearly and effectively public health laws and regulations
Advocate for the enforcement of laws and regulations pertaining to public health
Understand the administrative, social, and political implications of alternative policy options
Work with, coordinate, and/or lead community efforts to address public health problems
Build strong and ongoing relationships with the community
Interact, inform, and educate individuals from diverse backgrounds
Use the media and other forums to inform, educate, and empower people about health issues
Collaborate with colleagues and the community to manage and investigate public health problems
Accurately and effectively communicate information to a professional and a lay audience
Lead and participate in groups to identify public health problems
Use visual representations of data to identify public health problems
The results of the semi-structured interview showed that participants indicated a perception of high quality for the training methods, contents, presentations, instructors' responsiveness and value, and the sessions overall. Most participants (N = 37, 90.24%) thought these training methods were excellent or very good, and that made the training contents could be understood clearly and easily, and 85.37% (N = 35) of participants were satisfied with the trainers' performance. Analysis of results showed that more than ninety-five percent of participants (N = 39) reported that the training model was scientific and feasible. Additionally, most participants (N = 40, 97.56%) were very satisfied with the venue, training logistics and services, and only one participant (N = 1, 2.44%) suggested logistics and services should be improved.
Leadership response to avian influenza outbreaks
Previous evaluation of the leadership training program highlighted the participants' enhanced knowledge and changed attitude. Unexpectedly during the follow-up survey period, an outbreak of avian influenza occurred in Anhui and Hunan provinces, and two cases of human infections with avian influenza cases were confirmed, one of whom died. More than ten thousand poultries were infected and over fifty thousand poultries were killed [23, 24]. Some leaders then participating in the emergency preparedness training from the Anhui and Hunan were involved firsthand in the outbreak response, which included the organization, management, operations of emergency response, etc. Although the leadership training program had no plans to evaluate the scope and dynamics of this outbreak response, the effectiveness of the leadership training was shown indirectly. The competency of the leaders, who participated in the emergency preparedness training from the Anhui and Hunan provinces, was highly praised. By personal communication, participants' colleagues and superior leaders indicated that this emergency preparedness training not only improved the capability of public health leaders to perform their task efficiently and effectively, but also brought about attitudinal changes.
Studies showed that continuous education and training is a process of updating knowledge, developing skills, bringing about attitudinal changes, and improving the emergency competency of participants to perform their tasks efficiently and effectively [25, 26]. However, an effective and feasible training model is key to the success of an emergency training program. The leadership training program was designed strategically, which was an iterative and continually evolving process. As demonstrated by the feedback loops in the model (Fig. 1), the leadership training continuously strives to improve the system and enhance the preparedness of the public health leaders in responding to emergencies . First, the needs assessments were conducted by individual and organizational analysis, and based on this information, instructional objectives, specifying what is to be achieved in the training, were developed. Second, the curriculum was developed in consultation with key stakeholders and was modified to suit the leaders' backgrounds and project needs. Third, the interactive training methods might be helpful in increasing the quality of the training and improving retention of knowledge through immediate reinforcement of learning [27–30]. Fourth, the strategic definition of projects encouraged the integrated functioning of emergency responders from public health. Through training, the results showed that the curriculum could empower public health leaders to influence state-level policy makers concerning the organization, management, and operations of emergency response. The results of follow-up also showed a significant improvement in the knowledge levels and skills of leaders who participated in the training. This suggests that it was an effective and feasible training model.
In addition, we observed an important result in that the mean scores of emergency analytic/assessment knowledge increased instead of declining during the 12-month follow-up period. This is similar to results found by Qureshis KA et al . For this kind of phenomenon, we must take into consideration the experience of some public health leaders who participated in avian influenza outbreaks during the follow-up survey in 2007, which would provide practice and increase perceived relevance of the training and would likely positively affect training effectiveness. Nevertheless, the increased overall knowledge score and the positive change of skills suggested that training programs on emergency preparedness resulted in knowledge gaining and emergency capacity shifting. This is also similar to results found by Potter et al .
Study limitations should be noted. First, the sample frame construction was limited to the leaders that were from public health organizations/agencies, and did not include officials from other organizations/agencies. Second, evaluations were based on changes over time without the use of a horizontal comparison group. Finally, the composite variable created and used in the know-groups validity analysis may overestimate the effectiveness of training. Thus, it was not possible to fully determine which changes were due to the leadership training program and which were the result of other factors. These results, however, remained constant throughout, which provided evidence that these changes were due to the training program.
The leadership training model allows for a systematic approach to assessing, designing, developing, implementing, and evaluating training that is both competency-driven and practice-based. Furthermore, the model is a closed-loop system, and the continuous monitoring and evaluation could improve the effectiveness of training. In summary, the emergency preparedness training met its aims and objective at a satisfactory level, and improved the capability of public health leaders, and developed and delivered an effective and feasible leadership training model.
The emergency preparedness training program was supported by the World Health Organization. The authors thank all of the participants and trainers for their hard work, and all of the coordinators for their support and help. In addition, the authors would like to thank Sara Rohani and Mkangara OB for their critical reading of the manuscript.
- Gebbie KM: The public health workforce: key to public health infrastructure. Am J Public Health. 1999, 89: 660-1.View ArticlePubMedPubMed CentralGoogle Scholar
- Lowrey W, Evans W, Gower KK, Robinson JA, Ginter PM, McCormick LC, Abdolrasulnia M: Effective media communication of disasters: pressing problems and recommendations. BMC Public Health. 2007, 7 (147): 97-10.1186/1471-2458-7-97.View ArticlePubMedPubMed CentralGoogle Scholar
- Saleh SS, Williams D, Balougan M: Evaluating the Effectiveness of Public Health Leadership Training: The NEPHLI Experience. Am J Public Health. 2004, 94 (7): 1245-9.View ArticlePubMedPubMed CentralGoogle Scholar
- Umble K, Steffen D, Porter J, Miller D, Hummer-McLaughlin K, Lowman A, Zelt S: The National Public Health Leadership Institute: evaluation of a team-based approach to developing collaborative public health leaders. Am J Public Health. 2005, 95 (4): 641-4. 10.2105/AJPH.2004.047993.View ArticlePubMedPubMed CentralGoogle Scholar
- L Wright D, Rowitz L, Merkle A, Reid WM, Robinson G, Herzog B, Weber D, Carmichael D, Balderson TR, Baker E: Competency development in public health leadership. Am J Public Health. 2000, 90: 1202-7.View ArticleGoogle Scholar
- Kochar MS, Robertson RG, Mone MA: A faculty leadership development program at the Medical College of Wisconsin. WMJ. 2003, 102 (2): 24-8.PubMedGoogle Scholar
- Ottoson JM, Streib G, Thomas JC, Rivera M, Stevenson B: Evaluation of the National School Health Coordinator Leadership Institute. J Sch Health. 2004, 74 (5): 170-6.View ArticlePubMedGoogle Scholar
- Winnail S, Dorman S, Stevenson B: Training leaders for school health programs: the National School Health Coordinator Leadership Institute. J Sch Health. 2004, 74 (3): 79-84.View ArticlePubMedGoogle Scholar
- Potter MA, Burns HK, Barron G, Grofebert A, Bednarz GD: Cross-sector leadership development for preparedness. Public Health Rep. 2005, 120 (1): S109-15.Google Scholar
- Buehler JW, Whitney EA, Berkelman RL: Business and public health collaboration for emergency preparedness in Georgia: a case study. BMC Public Health. 2006, 6: 285-10.1186/1471-2458-6-285.View ArticlePubMedPubMed CentralGoogle Scholar
- Wei S, Liao QH, Nie SF, Luo LQ, Hu SH: Investigation on capabilities of early warning, analysis and decision making for public health emergency. Chin J Public Health. 2007, 23 (3): 347-8.Google Scholar
- Report of the system construction on disease control and prevention. Hubei. 2005, 66-9.
- Goldstein IL, Ford JK: Training in organizations: Needs assessment, development, and evaluation. 2002, Pacific Grove (CA): Brooks-Cole, 4Google Scholar
- Sarpy SA, Chauvin SW, Hites LS, Santacaterina L, Capper S, Cuccia M, Anderson AC, Petersen D: The south central center for public health preparedness training system model: a comprehensive approach. Public Health Rep. 2005, 120 (1): S52-8.Google Scholar
- Goldstein IL, Gilliam P: Training system issues in the year 2000. Am Psychol. 1990, 45: 134-43. 10.1037/0003-066X.45.2.134.View ArticleGoogle Scholar
- Columbia University School of Nursing, Center for Health Policy: Bioterrorism & emergency readiness: competencies for all public health workers. New York: Columbia University School of Nursing Center for Health Policy. Supported by the Centers for Disease Control and Prevention/Association of Teachers of Preventive Medicine Cooperative Agreement #TS 0740. Accessed November 2008, [http://www.cumc.columbia.edu/dept/nursing/chphsr/projects/emerprep/pdf/Competencies.pdf]
- Gebbie K, Merrill J: Public health worker competencies for emergency response. J Public Health Manag and Pract. 2002, 8: 73-81.View ArticleGoogle Scholar
- Barron G, Glad J, Vukotich C: The use of the National Public Health Performance Standards to evaluate change in capacity to carry out the 10 essential services. J Environ Health. 2007, 70 (1): 29-31.PubMedGoogle Scholar
- Qureshi KA, Gershon RR, Merrill JA, Calero BA, Murrman M, Gebbie KM, Moskin LC, May L, Morse SS, Sherman M: Effectiveness of an emergency preparedness training program for public health nurses in New York City. Fam Commun Health. 2004, 27 (3): 242-9.View ArticleGoogle Scholar
- The Public Health Workforce: An Agenda for the 21st Century. 1997, Washington, DC: US Dept of Health and Human ServicesGoogle Scholar
- Hay RD, Anderson R, Revicki D: Psychometric considerations in evaluating health-related quality of life measures. Qual Life Res. 1993, 2 (2): 441-9.Google Scholar
- Nunnally JC, Bernstein IH: Psychometric theory. McGBAW-HILL, INC, Third
- Ministry of Agriculture P. R. China. Accessed November 2008, [http://www.agri.gov.cn/ztzl/fkqlg/yqfb/t20070521_818862.htm]
- Ministry of Health P. R. China. http://www.gov.cn/gzdt/2007-03/29/content_565813.htm. Accessed November 2008 &http://www.ahwst.gov.cn/dt2111111119.asp?docid=2111120701. Accessed November 2008
- Vathsala M: Wickramasinghe. Training objectives, transfer, validation and evaluation: a Sri Lankan study. Intel J Training and Dev. 2006, 10 (3): 227-47. 10.1111/j.1468-2419.2006.00256.x.View ArticleGoogle Scholar
- Petticrew MP, Semple S, Hilton S, Creely KS, Eadie D, Ritchie D, Ferrell C, Christopher Y, Hurley F: Covert observation in practice: lessons from the evaluation of the prohibition of smoking in public places in Scotland. BMC Public Health. 2007, 7 (147): 204-10.1186/1471-2458-7-204.View ArticlePubMedPubMed CentralGoogle Scholar
- Burke MJ, Sarpy SA, Smith-Crowe K, Chan-Serafin S, Salvador RO, Islam G: Relative effectiveness of worker safety and health training methods. Am J Public Health. 2006, 96 (2): 315-24. 10.2105/AJPH.2004.059840.View ArticlePubMedPubMed CentralGoogle Scholar
- Baldwin K, Lamantia J, Prozialeck L: Emergency preparedness and bioterrorism response: development of an educational program for public health personnel. Public Health Nurs. 2005, 22 (3): 248-53. 10.1111/j.0737-1209.2005.220308.x.View ArticlePubMedGoogle Scholar
- Quigley RJ, Taylor LC: Evaluating health impact assessment. Public Health. 2004, 118 (8): 544-52. 10.1016/j.puhe.2003.10.012.View ArticlePubMedGoogle Scholar
- Barnett DJ, Everly GS, Parker CL, Links JM: Applying educational gaming to public health workforce emergency preparedness. Am J Prev Med. 2005, 28 (4): 390-5. 10.1016/j.amepre.2005.01.001.View ArticlePubMedGoogle Scholar
- Qureshi KA, Gershon RR, Merrill JA, Calero BA, Murrman M, Gebbie KM, et al: Effectiveness of an emergency preparedness training program for public health nurses in New York City. Fam Commun Health. 2004, 27 (3): 242-9.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2458/8/377/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.