Skip to main content

Table 5 Critical appraisal of quantitative component of studies evaluating Indigenous community development projects (n = 10)

From: A systematic review of studies evaluating Australian indigenous community development projects: the extent of community participation, their methodological quality and their outcomes

1st author, year

Selection bias (A)

Study design (B)

Confounds (C)

Blinding (D)

Data collection methods (E)

Withdrawal & drop-outs (F)

Intervention integrity (G)

Analysis (H)

Summary rating

Mixed method studies (n = 8)

 Smith (2004) [53]

Moderate

Moderate

NA

Weak

Weak

Moderate

Collection of quantitative data stopped before real community action started.

Community-level allocation, individual-level analysis. No appropriate analysis of change in child growth over time.

Weak

 Lee (2008) [42]a

Weak

Weak

NA

Moderate

Weak

Moderate

Many youth involved in the interventions, no information on consistency, other community initiatives were running simultaneously (including stricter supply controls and rewards linked to school attendance).

Community-level allocation and analysis. Statistical methods described in other publication. Dates of data collection (2001–2004) do not line up with dates of intervention (2003–2005), no post-test data.

Weak

 Tyrell (2003) [43]a

Weak

Weak

NA

Moderate

Weak

Moderate

No description of who was exposed to the project and who weren’t, nor of possible external influences on outcomes.

Allocation on community and individual level. Evaluation on community, organisational and individual level. No statistical analysis (outcomes as percentages only).

Weak

 Guenther (2011) [58]

Strong

Weak

NA

Weak

Weak

N/A

All participants were part of the project; not all participants attended every session; it is likely that the results were influenced by other interventions put on the families.

Individual-level allocation and analysis; statistical analyses (frequencies and t-test) were appropriate; analysis performed on actual intervention status.

Weak

 Salisbury (1998) [44]a

Weak

Moderate

NA

Weak

Weak

Moderate

No description of exposure or consistency; no mention of other interventions influencing outcomes; tested for population growth (which didn’t grow)

Unit of allocation and analysis are on organizational level. No statistical analysis.

Weak

 Hunt (2010b) [59]

Weak

Weak

Weak

Weak

Weak

Weak

No description of exposure or consistency, potential influence of other interventions running in the communities at the same time.

Unit of allocation community and organisational level, unit of analysis individual level, no statistical analysis.

Weak

 Moran (2003) [45]a

Moderate

Moderate

NA

Weak

Moderate

Weak

92 % of participants reported awareness of town plan. Outcomes may be influenced by the cycle of optimism and pessimism.

Unit of allocation is community level, analysis is done on community and individual level. No statistical analysis.

Weak

 McCalman (2005) [54]

Moderate

Moderate

NA

NA

Weak

Weak

All evaluation participants were exposed to intervention; consistency was not measured; outcomes likely influenced by other factors.

Project was allocated at organizational level, data were collected on community level, cannot be sure whether changes at community level are caused by changes at the organizational level. No statistical analysis

Weak

Quantitative only studies (n = 2)

 Jarvie (2008) [46]a

Weak

Weak

Weak

Weak

Weak

Strong

One community withdraw temporarily, others stayed. There’s a chance that outcomes are influenced by other developments going on at the same time

No statistical analysis, intervention offered at community level, data gathered at population level.

Weak

 Shannon (2001) [47]a

Weak

Moderate

N/A

Weak

Weak

Moderate

No description of exposure to intervention or consistency in delivery. Outcomes may be influenced by other factors

Community-level allocation and analysis. Appropriate statistical analysis.

Weak

  1. Note: Appraised using the Dictionary for Effective Public Health Practice Project Quality Assessment tool [38]
  2. a Published in peer-reviewed literature