Integrating a framework for conducting public health systems research into statewide operations-based exercises to improve emergency preparedness

Statewide exercise

The Medical and Health Exercise (hereafter referred to as the “Statewide Exercise”) is one of two annual emergency preparedness exercises coordinated at the state level in California. During this multi-level, multi-agency exercise, all 58 operational areas^b and 6 regional offices in California are invited to participate in functional exercises in collaboration with state agencies using a single common emergency scenario [9]. Each operational area designs and conducts the exercise to suit its own preparedness goals and objectives, either conducting the exercise independently or in collaboration with other jurisdictions. Although exercise participation varies among operational areas, certain agencies such as hospitals, public health departments and emergency medical services (EMS) agencies are incentivised to participate to meet grant or regulatory requirements.

The primary aim of this paper is to describe the process of conducting research using statewide exercises. By presenting the results of two example research questions, we illustrate the type of data that can be collected using this approach, and highlight the opportunities and challenges for future research.

An epidemiologic exercise model

Our research framework is based on the premise that operations-based exercises can be described using epidemiologic concepts (Figure 1). As a first step in actualizing the research framework, we used a simplified version of this model, represented by the letters a, b and c. The exercise scenario (denoted by a) constitutes the system perturbation or exposure, which can vary with different exercise injects. Upon receipt of the perturbation and injects, the exercise participants or agencies (denoted by b) perform actions (responses or intermediate outcomes, denoted by c) that directly or indirectly result in final outcomes, such as injury, disease, disability, and death.

Using the epidemiologic exercise model (Figure 1) as a guide, various types of study designs (randomized controlled trial, cross-sectional, etc.) can be implemented in the right exercise setting. The research possibilities are enormous and have not yet been fully tapped. To provide an example, we describe our method of studying two research priorities identified by our practice-based Steering Committee, which is composed of decision-makers in state and local public health and medical agencies. The intent of engaging a practice-based Steering Committee was to develop research priorities and questions relevant to practice, and to facilitate research translation into policy or practice [11].

Study design and instrument

Using a cross-sectional study design, we conducted a web-based survey to evaluate four domains related to the Statewide Exercise. The two described in this paper are: (1) organizational capabilities and functions and (2) challenges to inter-organizational communications and information sharing. In the context of our epidemiologic exercise model, these domains represent two ways of characterizing of response activities, which are conceptualized as intermediate outcomes in the causal pathway from the exercise perturbation event to final outcomes (Figure 1).

Throughout its development, the survey instrument was periodically reviewed by the Steering Committee and pilot-tested by representatives from participating organizations. Subsequent modifications were made based on this feedback.

Measurements

Study population

To evaluate California’s medical and public health system during the Statewide Exercise, representatives from all local health departments (LHDs; n = 61), local EMS agencies (n = 31), and Regional Disaster Medical and Health Coordinator/Specialists (RDMHC; n = 6) were invited to participate in the post-exercise survey. Since contact information for hospital preparedness staff is not publicly available, we relied on the assistance of the California Hospital Association (CHA) to recruit all general acute care hospitals (n = 466) in California, which comprise 87% of all licensed hospitals in the state (n = 534).

During a one-month data collection period following the Statewide Exercise, survey invitations and three subsequent reminders were emailed using a web-based system (Qualtrics^©). Survey recipients were chosen based on their functional role in their organization: health officers for LHDs, administrators for local EMS agencies, and preparedness coordinators for hospitals. Whereas government agency representatives directly received invitations and reminders from the Principal Investigator, hospitals received such communications as a forwarded message from CHA.

Data management and analyses

Survey data were restricted by date range and analyzed using Stata 11 (StataCorp LP, College Station, TX). The response rate was calculated after limiting the dataset to respondents who partially or fully completed the survey, and removing duplicate responses—the result of having multiple respondents from the same organization or from the same region, in the case of respondents who were RDMHC. When duplicate responses were found, researchers used a pre-determined prioritization scheme based on survey completion status and target functional role for an agency type (e.g., health officers were the intended functional role for LHDs) to determine which response to include.

Quantitative analysis

For both research questions, analysis was restricted to data from respondents who met the following criteria: (1) partially or fully completed the survey, (2) indicated their agency participated in the Statewide Exercise using the common scenario, and (3) indicated their agency carried out an operations-based exercise.

In order to characterize the capabilities and functions activated in response to the exercise scenario, we used the frequency of reported activation to find the average number and range of capabilities for each agency type. Capabilities that were commonly activated for all agency types, as well as those characteristic of a specific agency type, were identified via graphical response profiles (Figure 2).

Respondent and organizational demographics

Our study sample includes 174 respondents who represent 35 LHDs, 24 local EMS agencies, 121 hospitals, and 5 RDMHC in the state, giving response rates of 57%, 77%, 26% and 83%, respectively ^e,f. Of 174 respondents, 144 met the inclusion criteria for analysis of the questions relevant to this paper. These represent 30 LHDs, 21 local EMS agencies, 103 hospitals, and 5 RDMHC, which constitute 49%, 68%, 19% and 83%, respectively, of the target populations.^g

The final sample includes responses from agencies located in 46 of the 58 counties in California (79%). All six Mutual Aid Regions were represented by responding LHDs, local EMS agencies, and RDMHC (at least 35% of agencies in each Region responded). The most common functional roles of respondents were health officers or health directors (39% of LHDs), preparedness coordinators (50% of LHDs, 33% of local EMS agencies, 42% of local EMS agencies within a LHD, and 90% of hospitals), and administrators (44% of local EMS agencies and 92% of local EMS agencies within a LHD).

Key capabilities and functions

In this research, we characterize the response to the simulated IED emergency by the number and type of response capabilities (conceptualized as intermediate outcomes in our epidemiologic exercise model; Figure 1, letter c) activated by each agency and the role of each agency (i.e., lead versus supporting). Figure 2 provides a profile of response capabilities activated during the Statewide Exercise. Two sets of capabilities are depicted: first, a set of core capabilities activated across all agencies in response to the exercise scenario; second, a set of agency-specific capabilities, which were more common among certain types of agencies. The average number of capabilities reported was highest for local EMS agencies within a LHD (mean: 9.8, range: 3–18), followed by RDMHC (mean: 8.6, range: 1–17), hospitals (mean: 6.7, range: 1–24), local EMS agencies (mean: 5.0, range: 2–7), and LHDs (mean: 4.8, range: 1–10).

As Figure 2 shows, the core capabilities commonly activated across all agencies consist of medical surge, medical supplies management and distribution, critical resource logistics and distribution, emergency operations center management, and intelligence gathering and recognition of indicators and warnings. In terms of agency-specific capabilities, both types of local EMS agencies and hospitals commonly activated emergency triage and pre-hospital treatment. LHDs and local EMS agencies within a LHD commonly activated emergency public information and warning. Lastly, RDMHC and local EMS agencies within a LHD commonly activated responder safety and health, environmental health, emergency public safety and security, and citizen evacuation and shelter-in-place.

For more detailed results on key capabilities commonly activated across agency types, see Additional file 1. Additional file 1: Table S2 in the appendix summarizes the proportion of each agency type that activated each function during the exercise.

More subjectively, we describe the response by asking whether agencies played a lead or supporting role during the Statewide Exercise. Two-thirds (67%) of local EMS agencies and local EMS agencies within a LHD stated they played a lead role, while the majority of LHDs, hospitals, and RDMHC indicated they played a supporting role (61%, 76%, and 60%, respectively).

Challenges and opportunities

We were only able to partially operationalize our research framework for this Statewide Exercise, describing the perturbation (exercise) and intermediate outcomes (response activities) for different agencies within the public health system. To fully leverage the Statewide Exercise for the purpose of answering preparedness questions, a randomized controlled trial should be conducted by randomly allocating injects to agencies during exercise play. For example, positive or negative injects can be randomly assigned to agencies to measure exposure-response effects. Conducting such a study would require organizational buy-in from the exercise planning group and exercise participants at an early stage of exercise design and would require researchers to be particularly attentive to factors that might confound the exposure-response effects, including the local implementation of the exercise scenario, characteristics of the community served, and the distribution of public health responsibilities within a community. Under the right conditions, the approach we have described can be used to demonstrate variations in performance outcomes, such as the speed, quality, and the equitable provision of public health emergency services, and ultimately, to identify factors associated with high performing health departments.

In our study, we narrowly defined California’s public health and medical preparedness system to include local public health and EMS agencies, regional medical and health disaster coordinators, and hospitals. Despite this conservative definition, the system is complex, and includes the 61 LHDs, 31 local EMS agencies, 58 operational areas, 6 disaster medical and health regions, and 534 hospitals. In many organizations, emergency preparedness functions are not a full-time or a primary job responsibility, and individuals may fill the preparedness role for multiple organizations within the same county (e.g., EMS and public health agencies) or for multiple counties. Additionally, operations centers are often multi-use facilities, serving Department Operations Centers and Regional Operations Centers (e.g., the RDMHC may fill the regional disaster medical health role in a facility shared with an EMS agency). The overlapping roles of preparedness professionals in this system presented notable challenges in recruitment and data analysis, underscoring the complexity of California’s medical and health system and the subsequent challenge of assessing differences across agency types, which are not always clearly differentiated. Until the system components are better understood, the use of a web-based survey may not adequately capture how response functions are organized and delivered within the public health and medical system.

Although the integration of a research framework within the Statewide Exercise program yielded advantages of a larger sample size at a relatively low cost, there are several limitations. First, although state agencies recommend that operational areas include certain common features in their exercise design, each jurisdiction has the autonomy to develop an exercise that best meets their needs. Consequently, there is wide variation in the characteristics of exercises conducted across the state. To control for this, we limited our analysis to respondents who performed an operations-based exercise using the designated exercise scenario, on the specified day. However, additional variation in the exercise design (including the size, scope, and location of the emergency scenario, and the level of exercise participation) could not be controlled for. As a result, it is not clear how much of the variability in the observed response is attributable to differences in local implementation. To improve the strength and validity of findings, we would recommend additional standardization of future exercise characteristics at the local level, as well as incorporating random allocation of exercise injects to agencies sharing similar characteristics in terms of organizational structure, community served, and public health responsibilities. However, we acknowledge that this might be a challenge, particularly because exercises of this scale involve many agencies, and health departments may not have control over important aspects of exercise planning and implementation.

Second, because all operational areas were encouraged to simultaneously conduct exercises on the same day, the number of IED attacks “experienced” by the system in a 24-hours period, while theoretically possible, is not realistic. Therefore, the communications issues experienced by agencies could have been caused by or exacerbated by this artificiality. Nevertheless, the burden placed on the system simulated in this scenario remains plausible. In a major earthquake scenario, it is possible there would be more communication bandwidth used, and more operational areas receiving, coordinating, and sending mutual aid than occurred during this exercise.

Third, while we were able to recruit a large number of hospitals (n = 121), the overall hospital response rate was 26%, much lower than that of other recruited organizations. We found that small hospitals were particularly under-represented. If the response functions activated and communications challenges experienced vary by hospital size, then our characterization of how hospitals responded to the exercise scenario may be skewed. Several factors may have contributed to the low hospital response rate. First, because contact information for hospital preparedness coordinators is not publicly available, we conducted recruitment through a third party. Compared to governmental agencies, we had more limited control over hospital recruitment efforts—we were not able to directly contact potential survey respondents and to tailor recruitment efforts for non-responding hospitals. Second, respondents were asked to identify a primary hospital affiliation; however, we found that some respondents served as preparedness coordinators for entire hospital systems. As a result of our assigning one hospital per respondent, we are likely to have underreported the number of hospitals that are actually represented in our sample.

Lastly, these research results only provide us with a profile of capabilities and a characterization of communications challenges related to an IED scenario. We expect that, given a different scenario, the capabilities activated and communications challenges experienced might differ in important ways. As part of a larger research program at UC Berkeley, these variations will be documented and analyzed in a wide range of exercise and real event scenarios. Future research will focus not only on whether capabilities were activated, but also how central these activities were to the agencies’ objectives.