This paper describes the application of a conceptual framework to a Statewide Exercise as a systems-based approach for studying public health emergency preparedness and response. Capitalizing on a pre-existing statewide operations-based exercise program, we were able to access a large population of medical and health agencies with minimal cost, and to characterize several aspects of the public health and medical response to the standardized emergency scenario. Our research findings may provide an evidence-based starting point for preparedness professionals’ dialogue about expected and actual organizational roles, responsibilities, and resource capacities within the public health system—an area that needs improvement, according the literature and our Steering Committee . Additionally, we identify specific challenges to inter-organizational communications and information sharing, which could provide specific areas for intervention.
To our knowledge, this represents the first attempt to develop and integrate a comprehensive research framework into the conduct of a statewide operations-based exercise. Attributes of this research framework that lend it unique strengths include: (1) having sufficient organizational units of analysis to test statistical hypotheses, (2) the ability to apply epidemiologic concepts to identify which public health system characteristics are associated with positive health outcomes, and (3) the ability to test hypotheses using randomized controlled trials in which agencies are randomly allocated different scenarios or injects during exercise play. Because this is a novel approach to conducting exercise-based preparedness research, in the following section, we describe the lessons learned and challenges faced.
Challenges and opportunities
We were only able to partially operationalize our research framework for this Statewide Exercise, describing the perturbation (exercise) and intermediate outcomes (response activities) for different agencies within the public health system. To fully leverage the Statewide Exercise for the purpose of answering preparedness questions, a randomized controlled trial should be conducted by randomly allocating injects to agencies during exercise play. For example, positive or negative injects can be randomly assigned to agencies to measure exposure-response effects. Conducting such a study would require organizational buy-in from the exercise planning group and exercise participants at an early stage of exercise design and would require researchers to be particularly attentive to factors that might confound the exposure-response effects, including the local implementation of the exercise scenario, characteristics of the community served, and the distribution of public health responsibilities within a community. Under the right conditions, the approach we have described can be used to demonstrate variations in performance outcomes, such as the speed, quality, and the equitable provision of public health emergency services, and ultimately, to identify factors associated with high performing health departments.
In our study, we narrowly defined California’s public health and medical preparedness system to include local public health and EMS agencies, regional medical and health disaster coordinators, and hospitals. Despite this conservative definition, the system is complex, and includes the 61 LHDs, 31 local EMS agencies, 58 operational areas, 6 disaster medical and health regions, and 534 hospitals. In many organizations, emergency preparedness functions are not a full-time or a primary job responsibility, and individuals may fill the preparedness role for multiple organizations within the same county (e.g., EMS and public health agencies) or for multiple counties. Additionally, operations centers are often multi-use facilities, serving Department Operations Centers and Regional Operations Centers (e.g., the RDMHC may fill the regional disaster medical health role in a facility shared with an EMS agency). The overlapping roles of preparedness professionals in this system presented notable challenges in recruitment and data analysis, underscoring the complexity of California’s medical and health system and the subsequent challenge of assessing differences across agency types, which are not always clearly differentiated. Until the system components are better understood, the use of a web-based survey may not adequately capture how response functions are organized and delivered within the public health and medical system.
Although the integration of a research framework within the Statewide Exercise program yielded advantages of a larger sample size at a relatively low cost, there are several limitations. First, although state agencies recommend that operational areas include certain common features in their exercise design, each jurisdiction has the autonomy to develop an exercise that best meets their needs. Consequently, there is wide variation in the characteristics of exercises conducted across the state. To control for this, we limited our analysis to respondents who performed an operations-based exercise using the designated exercise scenario, on the specified day. However, additional variation in the exercise design (including the size, scope, and location of the emergency scenario, and the level of exercise participation) could not be controlled for. As a result, it is not clear how much of the variability in the observed response is attributable to differences in local implementation. To improve the strength and validity of findings, we would recommend additional standardization of future exercise characteristics at the local level, as well as incorporating random allocation of exercise injects to agencies sharing similar characteristics in terms of organizational structure, community served, and public health responsibilities. However, we acknowledge that this might be a challenge, particularly because exercises of this scale involve many agencies, and health departments may not have control over important aspects of exercise planning and implementation.
Second, because all operational areas were encouraged to simultaneously conduct exercises on the same day, the number of IED attacks “experienced” by the system in a 24-hours period, while theoretically possible, is not realistic. Therefore, the communications issues experienced by agencies could have been caused by or exacerbated by this artificiality. Nevertheless, the burden placed on the system simulated in this scenario remains plausible. In a major earthquake scenario, it is possible there would be more communication bandwidth used, and more operational areas receiving, coordinating, and sending mutual aid than occurred during this exercise.
Third, while we were able to recruit a large number of hospitals (n = 121), the overall hospital response rate was 26%, much lower than that of other recruited organizations. We found that small hospitals were particularly under-represented. If the response functions activated and communications challenges experienced vary by hospital size, then our characterization of how hospitals responded to the exercise scenario may be skewed. Several factors may have contributed to the low hospital response rate. First, because contact information for hospital preparedness coordinators is not publicly available, we conducted recruitment through a third party. Compared to governmental agencies, we had more limited control over hospital recruitment efforts—we were not able to directly contact potential survey respondents and to tailor recruitment efforts for non-responding hospitals. Second, respondents were asked to identify a primary hospital affiliation; however, we found that some respondents served as preparedness coordinators for entire hospital systems. As a result of our assigning one hospital per respondent, we are likely to have underreported the number of hospitals that are actually represented in our sample.
Lastly, these research results only provide us with a profile of capabilities and a characterization of communications challenges related to an IED scenario. We expect that, given a different scenario, the capabilities activated and communications challenges experienced might differ in important ways. As part of a larger research program at UC Berkeley, these variations will be documented and analyzed in a wide range of exercise and real event scenarios. Future research will focus not only on whether capabilities were activated, but also how central these activities were to the agencies’ objectives.