An epidemiologic exercise model
Our research framework is based on the premise that operations-based exercises can be described using epidemiologic concepts (Figure 1). As a first step in actualizing the research framework, we used a simplified version of this model, represented by the letters a, b and c. The exercise scenario (denoted by a) constitutes the system perturbation or exposure, which can vary with different exercise injects. Upon receipt of the perturbation and injects, the exercise participants or agencies (denoted by b) perform actions (responses or intermediate outcomes, denoted by c) that directly or indirectly result in final outcomes, such as injury, disease, disability, and death.
Using the epidemiologic exercise model (Figure 1) as a guide, various types of study designs (randomized controlled trial, cross-sectional, etc.) can be implemented in the right exercise setting. The research possibilities are enormous and have not yet been fully tapped. To provide an example, we describe our method of studying two research priorities identified by our practice-based Steering Committee, which is composed of decision-makers in state and local public health and medical agencies. The intent of engaging a practice-based Steering Committee was to develop research priorities and questions relevant to practice, and to facilitate research translation into policy or practice [11].
Study design and instrument
Using a cross-sectional study design, we conducted a web-based survey to evaluate four domains related to the Statewide Exercise. The two described in this paper are: (1) organizational capabilities and functions and (2) challenges to inter-organizational communications and information sharing. In the context of our epidemiologic exercise model, these domains represent two ways of characterizing of response activities, which are conceptualized as intermediate outcomes in the causal pathway from the exercise perturbation event to final outcomes (Figure 1).
Throughout its development, the survey instrument was periodically reviewed by the Steering Committee and pilot-tested by representatives from participating organizations. Subsequent modifications were made based on this feedback.
Measurements
Capabilities and functions
As a proxy measure of organizational roles and responsibilities assumed during a response—areas previously identified as requiring further research and improvement [1]—we assessed the capabilities and functions activated by public health and medical agencies during the Statewide Exercise. Standard response capabilities and functions were introduced by the Department of Homeland Security Target Capabilities List, which defines activities and tasks that form the basis of performance metrics and benchmark criteria to assess preparedness levels [12]. The term capability refers to the ability to perform functions or activities necessary for an effective response to major disasters and emergencies. Survey respondents were asked to indicate whether persons in their organization were assigned to functions related to each of 33 capabilities. Common capabilities were excluded from the survey, since they are cross-cutting and expected to be engaged in every response. Psychological support was added based on researchers’ interests.
Inter-organizational communications & information sharing challenges
To identify challenges to inter-organizational communications and information sharing, the survey included two questions. The first was a subjective and open-ended question, asking respondents to describe their most significant communications challenge during the exercise. The second instructed respondents to select the types of communication challenges their organization experienced from a list of options that was developed from the literature and Steering Committee members’ experiences.
Exercise participation
Survey respondents were also asked about the types of exercises conducted during the 2010 Statewide Exercise, characteristics of participating agencies, and factors influencing exercise participation. This information has been published elsewhere [13].
Study population
To evaluate California’s medical and public health system during the Statewide Exercise, representatives from all local health departments (LHDs; n = 61), local EMS agencies (n = 31), and Regional Disaster Medical and Health Coordinator/Specialists (RDMHC; n = 6) were invited to participate in the post-exercise survey. Since contact information for hospital preparedness staff is not publicly available, we relied on the assistance of the California Hospital Association (CHA) to recruit all general acute care hospitals (n = 466) in California, which comprise 87% of all licensed hospitals in the state (n = 534).
During a one-month data collection period following the Statewide Exercise, survey invitations and three subsequent reminders were emailed using a web-based system (Qualtrics©). Survey recipients were chosen based on their functional role in their organization: health officers for LHDs, administrators for local EMS agencies, and preparedness coordinators for hospitals. Whereas government agency representatives directly received invitations and reminders from the Principal Investigator, hospitals received such communications as a forwarded message from CHA.
Data management and analyses
Survey data were restricted by date range and analyzed using Stata 11 (StataCorp LP, College Station, TX). The response rate was calculated after limiting the dataset to respondents who partially or fully completed the survey, and removing duplicate responses—the result of having multiple respondents from the same organization or from the same region, in the case of respondents who were RDMHC. When duplicate responses were found, researchers used a pre-determined prioritization scheme based on survey completion status and target functional role for an agency type (e.g., health officers were the intended functional role for LHDs) to determine which response to include.
Agency classification
We used survey respondents’ self-designation to classify agency types. Five types of respondents emerged—those who declared an affiliation with: (1) LHDs, (2) local EMS agencies, (3) LHDs and local EMS agencies, (4) hospitals, and (5) RDMHC. Follow-up interviews were conducted with respondents who identified an affiliation with both LHDs and EMS agencies, all of whom affirmed their responses reflected both agencies’ experiences during the exercise. Since these respondents were affiliated with agencies legally recognized as local EMS agencies that operate within a LHD (California Health and Safety Code Section 1797.200), this category is hereafter referred to as “local EMS agency within a LHD.”d
Quantitative analysis
For both research questions, analysis was restricted to data from respondents who met the following criteria: (1) partially or fully completed the survey, (2) indicated their agency participated in the Statewide Exercise using the common scenario, and (3) indicated their agency carried out an operations-based exercise.
In order to characterize the capabilities and functions activated in response to the exercise scenario, we used the frequency of reported activation to find the average number and range of capabilities for each agency type. Capabilities that were commonly activated for all agency types, as well as those characteristic of a specific agency type, were identified via graphical response profiles (Figure 2).
Qualitative analysis
Qualitative data were independently coded by two researchers (JH, MP) to classify statements into themes, categories, and sub-categories. All coding discrepancies were resolved. Descriptive summaries of themes are provided and supplemented by illustrative quotes (see Additional file 1). These analyses were further informed by direct observation of the exercise by researchers.
This research was approved by the Committee for Protection of Human Subjects at the University of California, Berkeley.