In this paper we reviewed Australian Commonwealth and State and Territory government health planning policy to find out what principles governments state as being important to the work they do in Aboriginal and Torres Strait Islander contexts. We then assessed whether these principles are reflected in evaluation commissioning and evaluation practice. We found that despite a strong rhetoric that emphasised the importance of partnership and holism and to a lesser degree cultural competency and capacity building, these principles were not widely reflected in evaluation practice for the period 2007–2017.
For Commonwealth and State and Territory governments, establishing a solid evidence base that centres Aboriginal and Torres Strait Islander perspectives and understandings across the policy cycle (planning, implementation, evaluation) were stated as goals of the work they do. We found that principles such as evidence-based, accountability and equity underpinned most of the evaluations. Despite all seven health plans including the principle of partnership approaches, there was imperfect evidence of this in practice. The mechanisms to centre Aboriginal and Torres Strait Islander perspectives and understandings to fulfil the principle of partnership were less frequently embedded in evaluation practice. We note that although evaluations often engaged Aboriginal people or communities (i.e., consulted with key people or had an Aboriginal reference group), engagement as a mechanism for including Aboriginal perspectives is not always as far reaching as partnerships where the ACCS is at least theoretically provided ownership of decision making. However, in saying this we remain mindful that the process of colonisation impacts the power dynamic between Aboriginal partner/s and any dominant institution, where economic, resourcing and political power is largely held by the later [28]. In health planning, the meaning of engagement encompassed concepts from consulting through to community control, but rarely did we see Aboriginal community controlled organisations leading or enacting the principle of self-determination over evaluation. Although not represented in government health planning documents, many have asserted the importance of embedding the principle of self-determination in evaluation so that Aboriginal perspectives can be centred [29, 30]. Examples of evaluations that centre self-determination principles in utilising the skills, strengths and knowledge of the community-controlled sector are rare in contemporary practice [29,30,31,32,33]. Indeed, NACCHO (2019) raises concerns that evaluations are largely dominated by a private sector that is spatially and culturally removed from Aboriginal people and organisations, and question how such practice can centre Aboriginal perspectives, knowledge and experiences [29].
Our finding that the principles of equity, accountability and evidence-based most frequently underpin evaluations is best understood in terms of wider Commonwealth policy. The 2007–2017 period was marked by the Closing the Gap policy agenda and its overarching objective to build a robust evidence base, addressing disparities and achieving health equity for Aboriginal and Torres Strait Islander people. In recent years there have been increasing criticisms that Closing the Gap policy lacks the perspective of Aboriginal people through the decade-long exclusion of key Aboriginal bodies such as NACCHO from leading policy formation [34]. Policy without the strong voice of the ACCS has meant that diverse aspirations of all Aboriginal people have been excluded in favour of policy whose objectives are only equitable for Aboriginal people who aspire to live under socio-cultural standards set by the dominant culture [35, 36]. The full breadth of aspirations of Aboriginal people can only be responded to when these voices formulate policy. The recent 2019 Closing the Gap Partnership Agreements signed by the Commonwealth Government, State and Territory Governments and the Coalition of Aboriginal and Torres Strait Islander Peak Bodies has been a recent development to increase ACCS participation and involvement in Closing the Gap policy [37]. While peak bodies have long advocated for Aboriginal leadership in defining policy, health bodies such as NACCHO have also called for Aboriginal definitions of successful programs and policy to be given weight, alongside traditional evidence-based approaches [29].
In addition to lacking Aboriginal perspectives, evaluations infrequently centred Aboriginal understandings of health, despite health planning documents stating that a holistic understanding of health should underpin the work being done. Rather than drawing on NACCHOs definition of health adapted from the National Aboriginal Health Strategy (1989) (see Table 2) [27], evaluations instead focused on individual and biological outcomes (i.e., change in disease rates, behaviours or knowledge). We note that even when a holistic concept of health was present it was not well interpreted. It was not uncommon for evaluators to alter the concept of holistic health and present dominant social determinants, or individualised mental health measures. These findings echo a review by Lutschini (2005) that reported that despite the holistic concept of health being centrally placed in policy and strategies, policy makers often lacked a coherent articulation of the concept, were often uncritical and unreflective in their use and interpretation of it and often altered the concept and constitutional element without justification [38]. In saying this, we recognise that in the absence of widely accepted and valid quantitative measures for holistic health and wellbeing, evaluating for the holistic concept of health is difficult [39]. But we also know of the increasing scholarship in recent years to capture concepts of holistic health using narrative methods and Indigenous methodologies [40].
Across the Australian health planning documents there were not always consistent articulations of what was meant by the principles of shared responsibility, cultural competence, engagement, partnership, capacity building, equity, a holistic concept of health, accountability, or evidence-based. There were also varying degrees to which these articulations engaged with Aboriginal understandings and preferred articulations of these concepts.
A limitation of our review is that the principles we are evaluating are those that governments state as important to the work they do. Although such principles would have been developed through consultation with the ACCS, they may not capture those principles that are most important to Aboriginal people and the ACCS. We are cognisant that in neo-colonial contexts, health policy, including evaluation, cannot be separated from the control and regulation of Aboriginal bodies [6]. There is, without a doubt, an imbalanced power dynamic in current policy that sees dominant cultures policing Aboriginal populations and controlling the health agenda. As such, improving evaluation practice requires more than just governments doing what they say they will. As recognised by the Productivity Commission, there is a real need for transformative reform. We highlight that some of the principles that the ACCS have highlighted are important to evaluation and policy more broadly. These include: self-determination, community control, rights based approached, Aboriginal-led, investment in Aboriginal capacity, strengthen-based, do no harm, ethical, effective, transparent, cultural continuity, recognition of systems inequalities, recognition of past colonising and culturally safe evaluations, community benefit, transformative/decolonising orientations, social justice, Aboriginal cultural and intellectual property rights, empowering, and recognising diversity. It is obvious that the ACCS want Aboriginal perspectives, experiences and understandings to be central to evaluation policy pertaining to Aboriginal and Torres Strait Islander people [29,30,31,32,33, 41]. In addition, the reference group for this project that included members of the ACCS emphasised the importance of capitalising on Aboriginal strengths and data governance and sovereignty [13].
We also recognise that a limitation of our evaluation is that we only provide a count of the principles that have been reported within publicly available evaluations put out for advertised tenders by Australian governments. It is not clear what level of activity we have failed to capture. Our method of identifying public tenders, has also meant that we do not capture the full breadth of evaluations tendered by Aboriginal community controlled and other organisations. We realise that we are not capturing how such principles relate to internal evaluations, including those done by Aboriginal organisations as part of continuous quality improvement, monitoring and safety activity. In addition, the focus on external independent evaluations means that the evaluators are less likely to be subject to disincentives to reporting poor or adverse outcomes than might be the case in other forms of evaluation.
For evaluations to centre Aboriginal perspectives and understandings, there needs to be greater involvement of Aboriginal and Torres Strait communities and organisations in the planning and implementation of programs, policies and services, not just evaluation [2]. This recognises that many of the terms for evaluation including the outcome measures are decided early in the process of planning a policy, program or service [35]. As rightly identified by Altman (2019), for any principle to be meaningfully embedded in an evaluation there needs to be recognition of it across the policy cycle and importantly at the planning stage, where the parameters of the evaluation are largely set [35]. For example, principles such as self-determination, social justice or anti-racism cannot be fully realised in evaluation if measures of programmatic success are pre-defined by government. To centre Aboriginal perspectives there needs to be mechanisms for Aboriginal and Torres Strait Islander leadership and ownership at all phases of the program planning and evaluation cycle. With recent development of the ‘Ngaa-bi-nya Aboriginal and Torres Strait Islander framework’ and the ‘Lowitja Evaluation framework to improve Aboriginal and Torres Strait Islander health’ there are now frameworks that talk to the ethical responsibilities of doing evaluation in Aboriginal contexts [11, 13]. Use of ethical frameworks that delineate the responsibilities of all parties in evaluation (including commissioners, evaluators, implementors, Aboriginal community, Aboriginal participants) have potential to improve evaluation practice.
The findings we report here also highlight problems of transparency in current tendering and evaluation practice. We found that despite tenders being publicly listed none of the tendering organisations had mechanism for tracking once it was let. We were only able to access 5% of tenders. It is a concern that there is no publicly available repository for tenders as this would provide a means to conduct quality assurance practices to improve practice. We are also cautious that our findings only relate to 5% of all tenders; as such we do not know the true extent to which tendering practice is reflective of health planning. It should also be recognised that the tendering process is one way to change practice. One straightforward mechanism to include Aboriginal perspectives in current practice is for all tenders to explicitly state that a partnership and engagement approach is a requirement for prospective evaluators. This would also be beneficial for capacity building and cultural competency criteria to be included if commissioners are to align their practice with their health planning. Government principles for working with Aboriginal and Torres Strait Islander people should underpin evaluation tendering selection and reporting.
In Aboriginal health, it is widely accepted that evaluation reports remain ‘on the shelf’ and do not inform the next iteration in the health policy cycle. However, we found here that many reports did not even make it ‘to the shelf’ as we found only 1 in 4 reports was publicly accessible [13]. This raises important questions regarding to what extent evidence from evaluation contributes to the next phase of planning in the policy cycle, when it is not easily available, especially to the ACCS. Given that governments place such strong emphasis on evidence-based policy and programs in Aboriginal and Torres Strait Islander health contexts, the evidence must be made available in order to close the policy-evaluation cycle. We propose that transparency and accountability across evaluation can be improved by ensuring public access to tender documents, evaluation reports and documentation of responses to evaluations.
