Skip to main content

Table 1 Qualitative and quantitative instruments for process, formative, and effectiveness evaluation in the SHIFT project

From: Safety and Health through Integrated, Facilitated Teams (SHIFT): stepped-wedge protocol for prospective, mixed-methods evaluation of the Healthy Workplace Participatory Program

Instrument

Description

Content

Administration Frequency

Process Evaluation

Mobile App, “HWPP Assistant”

Meeting assessments and time logs by DT and SC members

• DT members and co-facilitators: participationa, knowledge, satisfaction, team effectiveness, engagementa, value

• DT co-facilitators: confidence in facilitating; quality and usability of HWPP materials

• SC members: awareness of team developments, organizational disturbances, team effectiveness, value, engagement

• Log of project-related time (level of participationa)

After each meeting

Coach Notes; Research Assistant Notes

Systematic, qualitative observations of DT, SC, and joint DT/SC meetings

• Attendancea

• Team dynamicsa

• Quality of Design Team facilitation (DT only)

• Level of management support (DT only)

• Progress through the structured IDEAS intervention design processa

• Any difficulties with the IDEAS process; any adaptations made

After each meeting

Design Team Meeting Records

Person-time invested; Fidelity of DT meetings to HWPP protocol

• DT meeting datesa

• Time spent in meetingsa

• Number of attendeesa

• Agenda items covered from the protocol

After each meeting

Formative and Effectiveness Evaluation: Short and Medium-term Outcomes

IDEAS Worksheets

DT work record of progress on IDEAS steps (summary worksheet)

• Content of discussions and interventions at each step: knowledge, breadth of topics covered over timea

After IDEAS Step 5

Process Check-up

DT and SC members: periodic survey evaluating the HWPP

• Organizational support and engagementa

• DT and SC engagementa

• Program facilitation

• Intervention planning/implementation

After IDEAS Step 5, then every 6 months during coaching

Design Team Survey

DT survey on members’ attitudes and perceptions of HWPP

• Member engagementa; Involvementa

• Managerial support for DT interventions

• Shared DT/SC knowledge and understanding about root causes of safety and health concerns

• Quality of organizational communication

• Expected HWPP impact

• Any unplanned consequences observed (negative or positive)

After IDEAS Step 5, then every 6 months during coaching

Interim Champion & Facilitator Interviews

Semi-structured interviews for champions and co-facilitators

• Satisfaction regarding program implementation

• Quality of and trends in DT and SC effectiveness, engagementa, involvementa, communication and organizational support

• Expectations regarding impacts of the participatory program overall and from the interventions being designed and implemented by the teams

Every 6 months during coaching

Formative and Effectiveness Evaluation: Longer-term Outcomes

All-Employee Survey

Self-administered survey (paper and pencil) for all facility employees

• Self-rated health, body weight, health behaviors, perceived health needs, attitudes and beliefs

• Symptoms of burnout and depression, history of injury and chronic disease

• Location/severity of musculoskeletal symptoms, related functional impairment

• Perceptions of working conditions, physical and psychological job features, organizational culture and climate

• Knowledge of and perceived effectiveness of health and safety programs

Baseline & study end

Organizational Readiness Survey

SC members, DT co-facilitators, and union leaders: organization’s readiness to form participatory DTs and resources available to support the HWPP (45-item survey)

• Existing programs for safety, health, and well-being

• Current program approaches for safety, health, and well-being

• Current resources available for safety, health and wellbeing

• Resources and readiness for change initiatives to improve safety, health and wellbeing

• Resources and readiness for use of teams

• Teamwork in work groups

• Resources and readiness for employee participation

• Management communication about safety, health, and well-being

Baseline & study end

Leadership Interviews

Scripted, open-ended interviews for SC members, DT co-facilitators, union leaders

• Perceived workforce safety and health concerns

• Employee involvementa in, successes of, and challenges to the existing safety program

• Current strengths, opportunities, and obstacles to the HWPP within the facility

• Shared DT/SC knowledge and understanding about root causes of safety and health concerns

• Expected HWPP impact

• Any unplanned consequences observed (negative or positive)

Baseline & study end

Pre-Post Intervention Surveys

DT-administered surveys

• Topic-specific measures based on key performance indicators (e.g., reduction of workplace exposures, short-term changes in behavior). Workforce sample determined by scope of intervention.

Pre- and post-IDEAS interventions

  1. aprimary outcome