In this section, results addressing this study's two research questions are reported sequentially. The results begin with findings associated with examining the relevance of the WHO guidelines at a community level, using the ACCISS case study in Canada (Figure 1). Subsequently, results associated with comparing the relevance of the guidelines to the ACCISS with two institution-based data gathering systems utilizing the S-EDISP in China, and the Y-CHIRPP in Glasgow are presented (Figure 2).
Relevance of attributes at a community level
At a macro level, the guidelines relate to and align with the nature of the environment in which the ACCISS is intended to function as well as how the system was initially developed and subsequently adopted for implementation by First Nation communities. Juxtaposing ISS attributes and the types of evaluation with the four inter-related activities of surveillance found that the guidelines gravitate towards an epidemiologic perspective. At the same time, the juxtaposition highlighted the potential alignment and use of these components for application at a community level when considered from a public health perspective (Figure 1).
At a meso level, the WHO defined attributes of a good surveillance system, fluently and consistently link to some aspect of the data collection process. Furthermore, unless juxtaposed across the four inter-related activities of surveillance, the attributes predominantly focus on epidemiologic issues and methods (Figure 1). In particular, the attributes of simplicity, flexibility and reliability focus attention on the nature and quality of ISS instrumentation as well as the quality and reliability of injury data being collected. The attribute of reliability also distinctly brings attention to epidemiologic issues addressing the completeness of each record, accuracy in relation to correct coding and classification of data as well as the representativeness of data being collected. Further, specific methods are described to calculate and quantify reliability based on measures of sensitivity and positive predictive value (PPV), respectively addressing the degree to which injury events are detected and non-injury events are excluded from a system. In addition, security and confidentiality emphasize the importance of ensuring the prevention of harm to individuals through the safekeeping of records and information gathered.
At a micro level, the relevance of the WHO defined attributes of a good surveillance were examined in relation to the age and stage of the Secwepemc Nation injury surveillance project. Accordingly, the epidemiologic dimensions were assessed with a primary focus on data collection while public health related dimensions were linked to data analysis and interpretation. Based on these foci, evaluation data revealed important interrelationships between the attributes of a good surveillance system and each respective surveillance activity. For instance, simplicity from an end-user perspective considered whether activities related to analysis, interpretation, and dissemination were sufficiently straightforward to ensure that these activities could be undertaken by community-based staff. Similarly, flexibility considered data collection tools and processes as well as the ability to tailor data analysis, interpretation, and dissemination functions to the needs and interests of each project community. This pattern was evident across attributes.
Acceptability as an attribute considers the willingness of individuals to participate in the system as well as whether results are being achieved. From this viewpoint, the attribute of acceptability encompassed data collection as well as a range of additional factors when considered in relation to data analysis, interpretation, and dissemination. The relevance of acceptability in relation to the ACCISS was linked to numerous factors, each of which consistently interrelated to each respective attribute of a good surveillance system and surveillance activities. Some of these factors included: the affordability of the data management tools; positive hands-on learning experiences associated with seeing and working with injury data; the capacity to self-generate reports; the ability to produce data that was considered relevant in the day-to-day work of staff; and the capacity of each project community to independently respond to emerging injury issues.
Lastly, utility, timeliness, and sustainability focus primarily on the practicality of procedures, budget, the ability to generate timely information, and system maintenance placing emphasis on staff and the supporting agencies actively involved in performing data related functions. Attention to data dissemination, the use of knowledge, and factors related to sustainability is notably limited in the WHO guidelines. Accordingly, these attributes shift the balance of attention towards data management and away from data dissemination and end-users needs. The relevance of these latter attributes to the ACCISS was consistently and closely connected with meeting Secwepemc Nation end-user needs and program goals. From an administrative viewpoint, the attributes of utility, timeliness, and sustainability of the ACCISS were identified as important considerations in the decision to implement the system. Furthermore, the necessity and interest of the project communities to strategically direct the use of limited resources was closely linked to overall acceptability of the system for implementation.
Ambiguity of recommended types of evaluation
The application of the three recommended types of evaluation presented challenges. Initial interpretations of this component were hindered by the choice of terminology and general lack of conceptual clarity. Despite this lack of conceptual clarity, two of the three recommended types of evaluation were notably consistent with an epidemiologic perspective. Retrospective evaluation was described in the guidelines as a process of looking back at injury records, using random selection strategies to examine and assess sensitivity and PPV. This description, focuses attention on epidemiologic issues, and typifies evaluation methods. The second type of evaluation, referred to as a process evaluation, was described as involving the direct observation and assessment of data records and data collection processes. This contracted explanation typifies evaluation methods with an epidemiologic focus rather than a comprehensive process evaluation. The third type of evaluation, a system environment evaluation, was described as assessing how well staff are able to operate the system. This description, which theoretically relates to looking at ISS operations and operational environments, was interpreted as being more representative of a category of factors to be considered for evaluation rather than a type of evaluation. This third type of evaluation was identified as being more consistent with a public health perspective.
Potential misalignment of guidelines for community-based application
Terminology aside, the attributes of a good surveillance system as defined by the WHO, types of evaluation, and associated methodologies, highlighted three areas of potential misalignment for community-based application. First, the attributes at a community level are subject to local interpretation and contingent on locally defined processes. Although the attribute of reliability, was identified as a very important construct for the Secwepemc Nation project communities, it was correlated with ensuring that data collection networks adequately reported and represented injury cases relative to each community population. As such, the efficacy of using sensitivity and PPV as metrics of reliability at the community level was determined to be very low by the ACCISS evaluation team. Based on the ACCISS the measures of reliability, outlined in the guidelines, represented a gap for community-based relevance and application (Figure 1).
In addition, the capacity building focus of the ACCISS centers attention on each of four inter-related injury surveillance activities of data collection, analysis, interpretation, and dissemination. As such, each attribute requires consideration relative to each of these four key activities. Moreover, since the use of injury data for injury prevention is central to the Secwepemc Nation project communities, the attributes of acceptability, utility, timeliness, and sustainability assume a higher level of relevance from a public health perspective. Overall, the relevance and interpretation of each of these attributes was driven by and linked to the end-users of the ACCISS.
Second, typical methodologies associated with an epidemiologic perspective are likely to be misaligned with community-based operational settings including culturally related factors (detailed in a separately drafted manuscript). One key factor related to the feasibility of evaluation methods for the ACCISS considered the number of injury cases documented relative to given time-periods, as well as the number and location of data collection sites and networks across a vast geographic territory. Given that the number and pattern of documented injury events varies considerably in relation to the size, location, and seasonal activities of each project community, field observations would have necessitated either multiple site-visits or residing in the communities for extensive periods. Issues of practicality, time and available resources outweighed any significant benefits.
At a community level, ACCISS data quality linked to missing or incomplete data, missing or duplicate cases and coding, is assessed and validated by way of a systematic and periodic audit of injury records by each project community. Audits were consistent with methods associated with the retrospective examination of records, however, simplified assessment criteria, standards, and baseline measures were being used for ongoing monitoring. From a cultural perspective, data collection involving third party observations of staff interviewing injured community members would have been intrusive and breached community-developed protocols and policies established to ensure confidentiality and anonymity. Evaluation methodologies as outlined in the WHO guidelines were only partially aligned with the operational environment of the ACCISS and represented a gap for community-based relevance and application (Figure 1).
Third, the relevance of the guidelines required consideration in view of the age and stage of the ACCISS as a program within the project communities. The nature and level of attention to the epidemiologic dimensions of the attributes shifted during pre-implementation and implementation stages. More time and emphasis was placed on monitoring the epidemiologic dimensions of the attributes during the early stages of implementation and less time afterward as data collection activities became more established. As data collection activities stabilized and became more routine, attention increasingly shifted towards the public health dimensions of data analysis, interpretation, and dissemination to promote use of injury data by end-users. This was reflected in the ACCISS evaluation, which identified early successes associated with the applied use of injury data. Project communities were using their data to engage community members and to address environmental risk factors identified in the community. This was found to be consistent with the high value that project communities placed on having community activities driven and informed by community data and represented a critical aspect related to overall acceptability of the system.
The application of the WHO guidelines for evaluation at a community level was enhanced by considering the attributes of a good surveillance system from the viewpoint of end-users in relation to the: goals and objectives of the ISS across inter-related injury surveillance activities; local operations and environments; as well as the program implementation stages of the ACCISS. Further, the attribute of acceptability when considered from the point of view of stakeholders and end-users served as an underpinning attribute across epidemiologic and public health dimensions (Figure 1). This facilitated balanced attention between data management issues and public health interests from an end-user perspective.
Relevance across different operational environments
The aim of reviewing two other ISS evaluations in relation to the ACCISS was to explore the application of the WHO guidelines across different operational settings and evaluation foci. The analysis found that although ISS are connected by a common interest to contribute to injury prevention and injury surveillance functions that basic characteristics across ISS and evaluation needs can vary significantly in relation to the location, scope, age, and locus of operational management of a system (Figure 2).
The S-EDISP in China and the Y-CHIRPP in Scotland were both Emergency Department and institution-based systems [10, 11]. The former encompassed multiple data collection sites gathering data on all injuries while the latter ISS was comprised of a single site collecting injury data specific to children. The ACCISS, as a community-based ISS, collected data on all injuries involving community members and non-residents injured within community boundaries. Like the S-EDISP, data collection involving the ACCISS encompassed multiple sites and locations. The S-EDISP was administered by a research center, the Y-CHIRPP by a hospital dedicated to the treatment of children, and the ACCISS by a First Nations Health Directors program team.
Relevance of age and status across ISS
Unambiguously, the focus of each evaluation was linked to the age and status of the ISS at the time each evaluation was conducted. Correspondingly, the application and interpretation of the WHO guidelines varied in relation to the age of each system. Conversely, interpretations of the guidelines were correlated with the level of attention and importance allocated to managing data collection, analysis, interpretation, and dissemination functions.
The S-EDISP evaluation focused on epidemiologic issues primarily centered on data collection activities [10]. The WHO defined attributes of a good surveillance system, as markers of performance focused predominantly on the feasibility of data collection. Reported results were concentrated on the system's instrumentation, case identification, data collection processes, and factors influencing data collection. Data analysis and interpretation functions were noted only briefly as being carried out by trained research staff while data dissemination was limited to a discussion about responses generated from a single question included on a self-administered staff questionnaire.
The ACCISS evaluation focused on pre-implementation and implementation phases of the project incorporating both epidemiologic and public health perspectives [9]. The attributes of a good surveillance system considered all key ISS functions from the viewpoint of the system's key stakeholders and end-users. Reported results were centered on project planning, implementation, ISS management and project outcomes as well as the capabilities and usefulness of the ACCISS in relation to data collection, analysis, and interpretation. Data dissemination functions were in the early stages of development at the time of evaluation.
The retrospective evaluation of the Y-CHIRPP, focused on an extensive program period from the program's inception to its conclusion [11]. Like the ACCISS, the Y-CHIRPP evaluation considered both epidemiologic and public health perspectives. The public health perspective emerged by way of an examination of the strengths and weaknesses associated with the steps of developing an ISS. Key individuals involved in these steps over the Y-CHIRPP program period informed this examination. Closely aligned with the four key functions of surveillance, reported results highlighted a significant and weighted emphasis on data collection with little to no emphasis directed towards data analysis, interpretation, and dissemination.
General relevance of guidelines across ISS
At a macro-level, given the general purpose and functions associated with ISS, the guidelines demonstrated a level of universal relevance for evaluation across three diverse operational environments. At a meso-level, the significance of the guidelines was less categorical. The relevance of the guidelines appeared contingent on the interpretation and use of the guidelines in relation to the age and status, locus of operational management, and design of the system. At the micro-level, relevance was dependent upon promoting congruency between performance measures and methods and system specific characteristics, processes, and operational environments.
Relevance of ISS characteristics to evaluation frameworks
System specific characteristics informed and influenced the focus and methods of evaluation associated with each ISS that was studied. The varied interpretation and application of the guidelines underscored the scope of considerations that can emerge relative to levels of multiplicity, acceptability, locus of operational management and differing perspectives.
Levels of multiplicity associated with ISS characteristics were evident within and across systems, in relation to such basic traits as the number of sites and partners associated with a system, number and type of individuals engaged in key surveillance functions, and system and site-specific structures and processes. These variations, whether community or institution-based, give emphasis to the symbiotic nature of ISS characteristics relative to each operational environment. At the same time, the multiplicity of ISS characteristics brings attention to how differently evaluation metrics are being interpreted and applied across diverse systems and settings.
Acceptability as an underpinning attribute
Acceptability, interpreted and conceptualized more broadly as an attribute underpinning both epidemiologic and public health perspectives, further highlights a range of inter-related ISS factors to be considered. Acceptability conceptualized in this way, in relation to the S-EDISP, Y-CHIRPP and ACCISS, was linked to a range of factors. Factors related to staff included: staffing levels and workload; staff support, motivation and perceptions; time limitations; and levels of training. Factors related to injury surveillance functions included: confidentiality; operating costs; levels of complexity and resources associated with each injury surveillance function; and the capacity to tailor and manage flexible processes [9–11]. Other key factors were related to: the ability to evaluate the reach and use of data collected by ISS; leadership support and continuity; involvement in decision-making processes; visible successes; measureable achievements; and the capacity to recognize, involve, and meet end-user needs [9–11].
The influence associated with the locus of operational management
In addition, the locus of operational management emerged as an important ISS characteristic influencing the development, management, monitoring, and evaluation of a system. In line with this characteristic, arise evaluation issues related to the historical background and origins of a system and the basis for its establishment. This background inherently inter-connected with factors related to leadership, organizational structure, mandate, and authority linked to the operational management of a system. Additional inter-related factors associated with the locus of operational management included: decision-making and accountability processes and structures; capabilities to manage all four key functions of injury surveillance; and the capacity to manage and coordinate collaborations involving partners and end-users [9–11]. In a related manner, the compatibility and deployment of epidemiologic and or public health perspectives appears contingent on operational environments.
The scope of issues identified relative to the evaluation of ISS is significant and the standard application of measures and methods of evaluation present challenges across different operational settings. Systematic attention to evaluation issues and methods, however, may support a starting place for re-conceptualizing evaluation frameworks.