Understanding Evaluation for Equity Programs: A guide to effective program evaluation
Type of Publication: Research report
Lead Organisation: NCSEHE
Year Published: 2015
Lead Researcher: Ryan Naylor
Written by Dr Ryan Naylor, Melbourne Centre for the Study of Higher Education, The University of Melbourne
EXECUTIVE SUMMARY
This guide is intended to help equity practitioners — particularly those who lack experience or confidence in program evaluation — to develop effective evaluation strategies for their programs. It is intended to complement existing institutional guidelines on evaluation, and acts as an initial introduction rather than a definitive guide to the subject.
Measuring the impact of equity programs is essential. Every program can benefit from having an evaluation plan, and evaluation should be a part of equity practitioners’ normal business. Evaluation is an opportunity to highlight the strengths and innovations of a program, as well as to identify and address weaknesses. Its primary purpose is to help you maximise your program’s effectiveness, not as an imposed performance management structure.
Evaluation involves gathering, reviewing and reflecting on information about your program. It follows a four step process, of establishing criteria; constructing standards; measuring performance and comparing with standards; and synthesising and integrating evidence into a judgement of worth. This same process applies whether you are evaluating a project plan, a pilot, or a fully operational project.
The most important aspect of evaluating a program is that it has clear, unambiguous and measurable aims and objectives. The program aims determine what evaluation questions are asked at each phase of the program’s life; which evaluation methods are appropriate to gather evidence to answer those questions; and which indicators and measures should be captured as evidence.
Appropriate alignment between program aims, evaluation criteria, methods and evidence gathered is essential. Having a detailed evaluation plan can be really useful in any program, to ensure this alignment is achieved. There are no easy answers to evaluating equity programs with rigor and detail. Typically, using a mixed methods approach is the one most likely to provide a full understanding of an equity program.
Evaluation should not be a burden. A clear evaluation plan will ensure data is gathered methodically, at time-appropriate intervals, and without wasting time gathering unnecessary data. The purpose of this guide is not to insist that practitioners collect more information.
Equity programs will benefit from formal or informal evaluation throughout their lives. As programs mature, the sorts of evaluation questions being asked should change. Evaluation during the planning phase may strengthen a project by clarifying its aims and improving the alignment between its aims and design, or with institutional targets. It may also establish criteria and standards for evaluation during later phases. Monitoring the program during its operation ensures the program is achieving its aims, and is being delivered effectively and consistently across multiple sites or practitioners. Finally, evaluation during the summative phase seeks to establish which aspects of the program worked and why, and what its impact has been. To fully capture the impact of your project, examining both unintended and intended outcomes can be very useful.
This guide is divided into 3 major parts.
1. Introduction
How do you evaluate an equity program?
This section of the guide provides a brief description of the major concepts in evaluating equity programs, and how to design an evaluation plan for a program.
2. When Do You Evaluate An Equity Program?
The next 3 sections provide resources to develop effective evaluation strategies at each of the 3 main phases of an equity program:
1. The planning phase;
2. Monitoring;
3. The summative phase.
3. Further Resources
This section provides references for further reading, and some questions and prompts to assist you in planning specific programs.
Read more: Understanding Evaluation for Equity Programs – A guide to effective program evaluation
This report is one of a series of 12 funded by the NCSEHE’s 2014 Student Equity in Higher Education Research Grants Program.