Strengthening evaluation in Indigenous higher education contexts in Australia
NCSEHE 2017 Equity Fellowship Final Report
James Smith
Charles Darwin University
Executive Summary
This report aims to unpack ways to strengthen evaluation in Indigenous higher education contexts in Australia. It is based on the outcomes of a 2017 Equity Fellowship funded by the National Centre for Student Equity in Higher Education (NCSEHE), and hosted through the Office of Pro Vice-Chancellor — Indigenous Leadership at Charles Darwin University (CDU). The Equity Fellowship was undertaken by Professor James Smith, with the support of Kellie Pollard, Kim Robertson and Fiona Shalley. An Expert Project Advisory Group was established to guide the direction of the project from the outset.
The Review of Higher Education Access and Outcomes for Aboriginal and Torres Strait Islander People (Behrendt, Larkin, Griew, & Kelly, 2012) provided a clear mandate for investing in policies and programs that support Indigenous pathways, participation and achievement in higher education in Australia. While there have been notable investments and significant national reforms in Indigenous higher education over the past few years, the recommendation within this report to develop a monitoring and evaluation framework is yet to be actioned. Similarly, in 2015, prior to its abolishment, the Aboriginal and Torres Strait Islander Higher Education Advisory Council (ATSIHEAC) recommended the development of a ‘performance framework’. This has also remained unactioned. As such, there remains minimal publicly available evaluation evidence in this space. In particular, there is scant evaluation evidence about program and policy effectiveness — that is, what does or does not work and why.
Interestingly, a similar trend has been noted in the broader Indigenous affairs landscape in Australia. Concern has consistently been raised about the lack of quality evaluation evidence generated through Commonwealth and philanthropically funded Indigenous affairs programs (Hudson 2016). As such, the Productivity Commission (2013), Australian Government Department of Prime Minister and Cabinet (2017) and the Australian National Audit Office (2017) have all acknowledged the importance of strengthening evaluation in Indigenous program and policy contexts across Australia, including that relating to higher education.
This report brings these two national conversations together. It begins by providing a snapshot of Indigenous higher education participation and achievement in Australia. This provides a background as to why evaluation in this context—which draws on Indigenous perspectives—is important now, more than ever. We then draw on recent academic scholarship and grey literature to discuss what we currently know about evaluation in Indigenous higher education.
The report then presents our findings from a qualitative research study involving 38 individual interviews and one group interview with two participant groups — (a) Indigenous scholars within higher education institutions; and (b) government policymakers with a role in equity and/or Indigenous higher education program and policy development and reform. The study asked questions about the current challenges and opportunities associated with undertaking evaluation in higher education contexts; the enablers and barriers associated with using evaluation evidence in policy and programs contexts; and ways to strengthen evaluation moving forward. We have deliberately privileged the voices of Indigenous scholars through this research process, as a commitment to valuing Indigenous worldviews and expertise, and in promoting concepts associated with data sovereignty. We demonstrate the utility of using this approach in evaluation research of this nature. Narratives from individual interviews with policymakers have, at times, been interwoven with those of Indigenous scholars. Efforts were made to undertake the research in consultation with key national stakeholders and peak bodies involved in Indigenous higher education, such as the National Aboriginal and Torres Strait Islander Higher Education Consortium (NATSIHEC).
Our research findings indicate that three key themes emerged from the thematic analysis:
1. Conceptualising ‘evaluation’ as a broad term
2. Towards a greater appreciation of qualitative methodologies and evidence
3. Towards greater accountability
Our research also identified 14 key enablers and drivers of evaluation in Indigenous higher education contexts. These included: growing Indigenous leadership; increasing funding and resources; investing in strategy development; leading innovative policy development, implementation and reform; investing in cultural transformation, change and quality improvement; addressing white privilege and power; improving Indigenous student outcomes; valuing Indigenous knowledges and prioritising Indigenous epistemologies; incentivising cultural competence; embracing political challenges as opportunities; promoting cultural standards and accreditation; reframing curricula to explicitly incorporate Indigenous knowledges and practices; investing in an Indigenous workforce; and recognising sovereign rights.
Through our research analysis, we demonstrate that these key enablers and drivers of evaluation in Indigenous higher education are primarily related to one of three domains of control — Indigenous control, government control; and university control. We argue that moving towards a greater synergy between these domains of control is important for strengthening evaluation in Indigenous higher education contexts in Australia. In some instances, privileging Indigenous control would be particularly advantageous.
Using our research findings, and in response to participant requests, we developed a conceptual model of potential performance parameters to strengthen Indigenous higher education monitoring and evaluation in Australia (presented on page 76). The model has four distinct spheres, each relating to the other. These include:
- students
- families and communities
- schools and other organisations
- universities.
We envisage the conceptual model can be used in multiple ways by practitioners, policymakers and researchers working in Indigenous and/or equity higher education contexts. For example, the model can be used as a preliminary benchmark to examine what is currently being collected and used to monitor progress; to identify current gaps in monitoring and evaluation to drive future national monitoring, performance and evaluation discussions; to visually represent potential performance parameters in a format that can be easily digested by key stakeholders; to demonstrate that potential performance parameters span multiple micro and macro levels; and to provide baseline information that could inform the development of a National Indigenous Higher Education Performance and Evaluation Strategy.
Seventeen recommendations for strengthening evaluation in Indigenous higher education are presented in this report. These recommendations span research, policy and practice contexts, and often sit at the nexus between them. In parallel with recommendations from previous national reports, we conclude by suggesting that the development of a National Indigenous Higher Education Performance and Evaluation Strategy is urgently required to advance Indigenous student outcomes in Australia.
Recommendations
- The Australian higher education sector, National Aboriginal and Torres Strait Islander Higher Education Consortium, Universities Australia and the Australian Government prioritise the development of a National Indigenous Higher Education Performance and Evaluation Strategy. This should be Indigenous-led and appropriately resourced.
- The Australian Government should include a suite of Indigenous higher education targets, aligned with a National Indigenous Higher Education Performance and Evaluation Strategy, as part of the Closing the Gap refresh.
- The National Centre for Student Equity in Higher Education and the National Aboriginal and Torres Strait Islander Higher Education Consortium co-host a national summit about evaluation in Indigenous higher education contexts. This should be used to strategically discuss the scope and nature of evaluation priorities and to map key areas for action. This should be Indigenous-led and appropriately resourced.
- Co-develop a glossary of terms associated with evaluation in higher education in consultation with key stakeholders including Indigenous scholars, policymakers and practitioners to ensure diverse viewpoints are captured.
- Stories and narratives are explicitly incorporated into reporting and evaluation processes examining the impact and outcome of Indigenous higher education. They provide a legitimate, culturally relevant and contextual source of evidence.
- The Australian Government explicitly incorporates qualitative reporting and evaluation processes into all higher education program funding agreements which aim to improve Indigenous higher education access and outcomes. This should complement existing quantitative data sets; and provide greater contextual information to inform future policy and program development and reform.
- Investment into the development of innovative qualitative evaluation strategies aligned with Indigenous methodologies and methods could provide new insights suitable for reforming Indigenous higher education policy and practice in Australia. This should be completed in consultation with Indigenous scholars.
- Accountability within Indigenous higher education contexts must be viewed as a shared responsibility between universities and government, and should involve both Indigenous and non-Indigenous stakeholders. The ‘community’ should remain the focal point in such discussions.
- A better and more visible harmonisation of communication and reporting processes associated with Australian Government policies and programs that support Indigenous higher education students and staff. This includes both Indigenous and equity-focused programs. Strategies which reduce working in silos within and between Australian Government Departments should be a priority.
- Clearly defined performance measures relating to the adoption whole-of-university approaches to Indigenous higher education should be embedded into all senior university executive contracts and reviewed regularly to increase individual accountability. Performance against these measures should be managed proactively, with clear consequences for poor performance.
- The Australian Government, NATSIHEC and the TEQSA should work collaboratively to expand the scope of Indigenous-focused higher education accreditation standards to increase university accountability.
- The Australian Government and philanthropic organisations mandate that a minimum of 10 per cent of all program funding in Indigenous higher education contexts is invested into evaluation; and that the Australian Government and universities are held to account against this mandated requirement, preferably through legislative change.
- The Australian Government make a dual and parallel investment in Indigenous capacity building focused on (a) evaluation knowledge and skill development; and (b) Indigenous leadership and governance, to increase Indigenous control in Australian higher education contexts.
- The NATSIHEC, Australian Government, universities, Universities Australia and other key stakeholders work collaboratively and strategically to invest in the 14 enablers and drivers identified in this report, with preference given to those associated with Indigenous control.
- The Australian Government recognises the sovereign rights of Indigenous peoples, as espoused in the UN Declaration of the Rights of Indigenous Peoples and the Coolangatta Statement on Indigenous Peoples’ Rights in Education by ensuring there are dedicated and appropriately resourced Indigenous education policy and program units in government departments, separate to those associated with equity funding.
- Use the conceptual model as a baseline for developing strategies and actions associated with the development of the NIHEPES.
- Conduct a meta-analysis of Australian research studies and evaluation reports examining Indigenous student and staff perspectives about pathways, transitions, participation, success and achievement in higher education.